A game agent design method and system based on deep reinforcement learning

Abstract

This invention relates to the field of game intelligent design technology, and discloses a game intelligent agent design method and system based on deep reinforcement learning. The method includes: collecting user data and analyzing user behavior, grouping and training users within these groups, performing cross-set integration training to form feedback-driven iterative training, identifying other games related to the current game, collecting and analyzing their data, grouping and training users across games, and performing final training after data fusion to generate a general intelligent agent. This game intelligent agent design method and system based on deep reinforcement learning collects comprehensive user behavior and feedback data, covering different types of data, ensuring that the intelligent agent can learn the diversity of user behavior. Through feature extraction and behavior pattern recognition, it deeply understands users' thinking patterns and gaming habits, effectively improving the performance and user experience of the intelligent agent, and ensuring its adaptability and competitiveness in different user groups and game scenarios.

Description

Technical Field

[0001] This invention relates to the field of game intelligent design technology, specifically to a game intelligent agent design method and system based on deep reinforcement learning. Background Technology

[0002] As an important form of entertainment, video games, with their complexity and diversity, provide a natural research platform for the development of artificial intelligence (AI). In recent years, with the rapid improvement of computing power and continuous innovation of algorithms, the application of artificial intelligence in the gaming field has made significant progress, especially in deep reinforcement learning. The rise of Deep Reinforcement Learning (DRL) has enabled AI to learn and master sophisticated strategies from complex interactive environments, thereby surpassing human and even professional players in various games. Traditional game AI design methods, such as rule-based systems, search algorithms (like Minimax), and finite state machines, while performing well in some simple games or specific situations, fall short when faced with the increasing complexity, massive state spaces, and dynamic, nondeterministic environments of modern games. These methods often require extensive manual design and tuning and are difficult to generalize to new games or different game scenarios. Deep reinforcement learning provides a powerful solution to these problems. DRL combines the powerful feature extraction capabilities of deep learning with the decision learning framework of reinforcement learning, enabling it to learn optimal strategies directly from raw game pixels or high-dimensional state spaces. Through extensive interaction with the game environment, the agent can continuously try and fail, and adjust its internal neural network parameters based on the reward signals obtained, ultimately learning complex behaviors that maximize cumulative rewards.

[0003] Existing game agent design methods and systems based on deep reinforcement learning cannot analyze game users' game thinking, group users with similar game thinking into the same set, train each user set first and then train the overall data, or group users of similar games according to their game thinking and train the agent according to the idea of ​​training groups first and then the whole data. This makes it easy for the agent to have repetitive or interfering data during the training process, which slows down the training process, reduces training efficiency, and limits its practicality. Summary of the Invention

[0004] This invention provides a method and system for designing game intelligent agents based on deep reinforcement learning, which helps to solve the problems mentioned in the background art.

[0005] This invention provides the following technical solution: a method for designing game intelligent agents based on deep reinforcement learning, comprising:

[0006] Data is collected from all users on the game server.

[0007] Perform user behavior analysis on all users on the game server;

[0008] Users on the game server are grouped and trained accordingly.

[0009] Based on the results of group training, perform cross-set integration training;

[0010] Collect user feedback on the trained agent and form feedback-driven iterative training based on the user feedback;

[0011] Identify other games related to the current game and collect and analyze the data;

[0012] Based on data collected and analyzed from other games related to the current game, cross-game user grouping and training are conducted;

[0013] By fusing all training data from the current game and related games and performing final training, a general intelligent agent capable of adapting to various user behaviors and habits is generated.

[0014] As part of the game agent design method based on deep reinforcement learning described in this invention, data collection is performed on all users in the game server, specifically:

[0015] For each user Extract its raw behavioral data from the game server, denoted as :

[0016] ;

[0017] Using feature extraction functions Transform raw behavioral data into low-dimensional feature vectors :

[0018] ;

[0019] Collect users Direct feedback data, denoted as : ;

[0020] The indirect feedback data is normalized to obtain and :

[0021] ;

[0022] ;

[0023] Integrating direct feedback and normalized indirect feedback, a comprehensive feedback data is formed, denoted as... :

[0024] ;

[0025] behavioral characteristics and feedback features By splicing together, we obtain the comprehensive features. : .

[0026] As a game agent design method based on deep reinforcement learning as described in this invention, the following is a specific method: User behavior analysis is performed on all users in the game server.

[0027] Obtain the feature vector of each user in the game server ;

[0028] Clustering the behavioral characteristics of all users to identify different behavioral patterns: ;

[0029] For each behavioral pattern Analyze and uncover the logic and rules behind user behavior:

[0030] .

[0031] As described in this invention, the method for designing game agents based on deep reinforcement learning includes: grouping users in the game server and training them accordingly, specifically:

[0032] S1. Randomly extract a user from the game server, designate them as the analysis user, and denot them as... ;

[0033] Randomly select another user from the game server and designate them as the comparison user, denoted as _____. ;

[0034] S2, Obtain and analyze users respectively Compared with users The eigenvectors of are denoted as . and ;

[0035] S3, Calculation and Analysis User Compared with users Cosine similarity:

[0036] ;

[0037] S4. Repeat steps S1-S3 until all other users in the game server have a cosine similarity value with the analyzed user.

[0038] S5. Repeat steps S1-S4 until all users in the game server have been analyzed as users.

[0039] S6. For each analytics user Treat it as a separate set;

[0040] S7, Extract and analyze users All corresponding comparison users ;

[0041] S8, Extract and analyze users Compared with all users Similarity matrix between :

[0042] ;

[0043] S9. Define a filtering function for each analysis user. Find all users with high similarity to it. :

[0044] ;

[0045] like Then determine the user analysis Compared with this user High similarity;

[0046] like Then determine the user analysis Compared with this user Low similarity;

[0047] S10, Analyze the user's judgment Compared with this user All users with high similarity Merge into the same set;

[0048] S11. Repeat steps S1-S10 until the similarity comparison of all users on the game server is completed, forming multiple user sets: ;

[0049] S12, for each set Initialize an agent : ;

[0050] S13. For each set Collect game behavior data and feedback data from all users within this set to form a training dataset. : ;

[0051] S14. For each agent Conduct independent training;

[0052] S15. Maximize the agent's performance in this user set environment:

[0053] ;

[0054] S16. Optimize the objective function and update the parameters of the policy network: ;

[0055] S17. Repeat steps S14-S15 until the agent's performance in the user set environment converges.

[0056] As a game agent design method based on deep reinforcement learning as described in this invention, the method includes: performing cross-set integration training based on the group training situation, specifically:

[0057] Get each collection Corresponding training dataset This will be integrated into a comprehensive dataset.

[0058] ;

[0059] Initialize a global agent : ;

[0060] Execution environment interaction;

[0061] After the game ends, use GAE to calculate the advantage function. : ;

[0062] Update the policy network by maximizing the objective function of PPO:

[0063] ;

[0064] Update the parameters of the policy network using gradient ascent: ;

[0065] Update the value function by minimizing the mean squared error: ;

[0066] Update the parameters of the value function using gradient descent: .

[0067] As described in this invention, the game agent design method based on deep reinforcement learning includes: collecting user feedback on the trained agent and forming feedback-driven iterative training based on the user feedback, specifically:

[0068] Select a group of users from all users on the game server. Conduct experience testing:

[0069] ;

[0070] For users Provide a game environment consistent with the training environment, and then train the agent. Deploy to the game environment;

[0071] user With intelligent agents Interaction, generating experience data : ;

[0072] For users Each user in Collect their direct feedback :

[0073] ;

[0074] Collect users The indirect feedback data is then normalized.

[0075] ;

[0076] ;

[0077] Integrating indirect feedback data : ;

[0078] Direct feedback and indirect feedback Concatenate into a comprehensive feedback vector : ;

[0079] Forming a comprehensive feedback dataset : ;

[0080] For users Each user in Comments Sentiment analysis is performed to identify positive and negative feedback and obtain a sentiment score. : ;

[0081] Calculate each behavioral feature Importance score : ;

[0082] For the comprehensive feedback dataset Perform clustering to obtain clustering results :

[0083] ;

[0084] Based on the feedback analysis results, define improvement goals. : ;

[0085] Select data relevant to the improvement goals from the original training data. :

[0086] ;

[0087] Using the improved dataset Targeted training of the agent and updating of the policy network and value function : ;

[0088] Update the improved policy network and value function to the global agent:

[0089] ;

[0090] ;

[0091] Repeated execution for users Each user in Collect their direct feedback The overall steps from updating the improved policy network and value function to the global agent are continued until the agent's performance reaches a satisfactory level.

[0092] As described in this invention, the method for designing a game agent based on deep reinforcement learning includes: identifying other games related to the current game and collecting and analyzing their data, specifically:

[0093] Analyze the current game's genre, gameplay, and user base to identify other games related to it: ;

[0094] For each relevant game To obtain user data separately: ;

[0095] For each relevant game Data is collected and user behavior is analyzed for each user in the process, resulting in... and ;

[0096] The user data from related games is integrated into the user data of the current game to form the final training dataset: .

[0097] As described in this invention, a game agent design method based on deep reinforcement learning is wherein: all training data from the current game and related games are fused and finally trained to generate a general agent capable of adapting to various user behaviors and habits, specifically:

[0098] Obtain the final training dataset ;

[0099] For the final training dataset Conduct cross-game user grouping and training.

[0100] The present invention also discloses a system for implementing a game agent design method based on deep reinforcement learning, wherein: it includes:

[0101] Data acquisition module: Collects game data from users of the current game and related games, and collects user feedback on the game and the trained agent;

[0102] Data Analysis Module: Analyzes game data of users of the current game and related games, classifies users, and groups users with similar game operation strategies into the same set for data training;

[0103] Data training module: Based on the user classification set of the current game or related games, first train the agent on the data within each set, and then train the agent as a whole according to the training status of the sets;

[0104] Data integration module: Integrates the training data of the agent for the current game and the training data of the agent for related games, and then trains the agent again;

[0105] Data generation module: Generates or updates the agent based on the training results.

[0106] The present invention has the following beneficial effects:

[0107] 1. This game agent design method and system based on deep reinforcement learning extracts training data from all users on the game server, sets up an in-game feedback mechanism to collect direct user feedback on the agent's performance, records indirect feedback indicators such as user game duration, frequency, and retention rate, preprocesses and extracts features from each user's game behavior data to extract key features that reflect user thinking patterns and habits, uses clustering algorithms to analyze user behavior features and identify different behavior patterns, and further mines the logic and rules behind user behavior through algorithms such as decision trees and random forests, defines a similarity measurement method between users, and calculates the similarity between users based on user behavior features and patterns. The system assesses the similarity between users, grouping them into multiple user sets. Users in each set share similar thinking patterns and gaming habits. Comprehensive user behavior and feedback data are collected, providing a rich information foundation for subsequent analysis and training. This data covers different types of data, ensuring the agent can learn from the diversity of user behavior. Through feature extraction and behavioral pattern recognition, the system gains a deep understanding of users' thinking patterns and gaming habits, providing a basis for subsequent personalized training. This allows the agent to better adapt to the needs of different users. By grouping similar users, noise during training is reduced, training efficiency is improved, and the agent can be optimized for the characteristics of different user groups, enhancing the user experience.

[0108] 2. This game agent design method and system based on deep reinforcement learning trains the agent independently on data from each user set using deep reinforcement learning algorithms. This allows the agent to adapt to the behavioral characteristics of users within that user set. During training, the agent's strategy is continuously adjusted to improve its performance within that user set environment. The training data from all user sets are then integrated into a comprehensive dataset. This integrated dataset is used to globally train the agent, enabling it to adapt to the behavioral characteristics of different user sets. Users then experience the trained agent, observing its performance during gameplay and collecting user feedback, including direct feedback (such as ratings and comments) and indirect feedback (such as changes in game behavior). This user feedback is analyzed to identify the agent's strengths and weaknesses. Based on feedback, the agent is trained in a targeted manner to optimize its performance in specific scenarios or behavioral patterns. It can learn the behavioral characteristics of specific user sets, providing a personalized gaming experience. Through independent training and strategy optimization, the agent's performance in specific user sets is improved. By integrating data from multiple user sets, the agent's generalization ability is enhanced, enabling it to adapt to a wider range of users. The agent's stability and robustness in different environments are strengthened. Through real user experiences and feedback, an objective evaluation of the agent's performance is obtained, providing direct improvement suggestions for subsequent iterative training, forming a closed-loop optimization. Targeted training based on user feedback solves specific problems, improves user satisfaction, and continuously optimizes the agent's performance to ensure its competitiveness in the face of ever-changing user needs.

[0109] 3. This game agent design method and system based on deep reinforcement learning identifies other games related to the current game, acquires user data from these games, analyzes the user data from other games to extract user behavior features and patterns, merges all training data from the current game and related games to form a final training dataset, and uses the merged dataset to train the agent, generating a general agent that can adapt to various user behaviors and habits. By collecting data from related games and expanding the training dataset, the adaptability of the agent is improved, enabling the agent to adapt to user behaviors in different game environments, enhancing its generalization ability. By integrating all data for final training, the agent's performance is ensured to be balanced in all aspects, generating a general agent that can adapt to various user behaviors and game scenarios, thus improving the overall user experience. Attached Figure Description

[0110] Figure 1 This is a flowchart of the game intelligent agent design method based on deep reinforcement learning of the present invention;

[0111] Figure 2 This is a system block diagram for implementing the game agent design method based on deep reinforcement learning according to the present invention. Detailed Implementation

[0112] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0113] Example 1: A method for designing game agents based on deep reinforcement learning (see [reference]). Figure 1 ,include:

[0114] Data is collected from all users on the game server.

[0115] Perform user behavior analysis on all users on the game server;

[0116] Users on the game server are grouped and trained accordingly.

[0117] Based on the results of group training, perform cross-set integration training;

[0118] Collect user feedback on the trained agent and form feedback-driven iterative training based on the user feedback;

[0119] Identify other games related to the current game and collect and analyze the data;

[0120] Based on data collected and analyzed from other games related to the current game, cross-game user grouping and training are conducted;

[0121] By fusing all training data from the current game and related games and performing final training, a general intelligent agent capable of adapting to various user behaviors and habits is generated.

[0122] Example 2 is an improvement upon Example 1. This deep reinforcement learning-based game agent design method collects data from all users on the game server, specifically as follows:

[0123] For each user Extract its raw behavioral data from the game server, denoted as The raw behavioral data refers to the user's specific operation records in the game, such as clicking buttons, movement trajectories, etc.

[0124] ;

[0125] in, This represents the user's behavior across different dimensions, which are the results of classifying the raw data according to dimensions such as operational, strategic, social, and achievement categories.

[0126] Using feature extraction functions Transform raw behavioral data into low-dimensional feature vectors :

[0127] ;

[0128] in, It was before The weight matrices of the principal components are used to project the original data onto the matrix formed by the previous components. In the low-dimensional subspace defined by each principal component It indicates that it is the former The transpose of the weight matrices of the principal components. This represents the standardized raw behavioral data. Standardization can make the mean of the data 0 and the standard deviation 1, thereby eliminating the differences in units between different features. Specifically:

[0129] ;

[0130] in, For each feature dimension The mean, For each feature dimension The standard deviation, and the feature dimension is each feature in the original behavioral data;

[0131] Collect users Direct feedback data, denoted as The direct feedback data refers to feedback actively provided by users, such as satisfaction ratings and questionnaire opinions. ;

[0132] The indirect feedback data is normalized to obtain and The indirect feedback data is derived indirectly through the analysis of user behavior, such as game duration, retention rate, and task completion rate.

[0133] ; ;

[0134] in, The normalized game duration The retention rate after normalization. and These are the maximum and minimum game times for all users, respectively. and These are the maximum and minimum retention rates for all users, respectively. and The game duration and retention rate before normalization are shown separately. After normalization, both game duration and retention rate will be normalized to the range of [0,1]. The retention rate refers to the proportion of users who use or participate in a service, game or product again within a specific time period. In the gaming field, it is usually used to measure the frequency with which players continue to visit the game after registration or first play. It is an important indicator for evaluating the attractiveness of the game and user stickiness.

[0135] Integrating direct feedback and normalized indirect feedback, a comprehensive feedback data is formed, denoted as... :

[0136] ;

[0137] behavioral characteristics and feedback features By splicing together, we obtain the comprehensive features. : .

[0138] This embodiment also provides user behavior analysis for all users on the game server, specifically:

[0139] Obtain the feature vector of each user in the game server ;

[0140] Clustering algorithms are used to cluster the behavioral characteristics of all users to identify different behavioral patterns: ;

[0141] in, It is the set of behavioral feature vectors of all users, denoted as , This is the clustering result, denoted as each This represents a behavioral pattern;

[0142] Use decision tree or random forest algorithms for each behavior pattern Analyze and uncover the logic and rules behind user behavior: ;

[0143] in, It is a behavioral pattern The corresponding set of feature vectors, These are the corresponding behavioral pattern tags. It is a behavioral pattern model that is constructed to understand behavioral patterns. The logic and rules governing user behavior;

[0144] This involves clustering the behavioral characteristics of all users to identify different behavioral patterns, specifically using the K-Means algorithm for clustering:

[0145] Random selection The eigenvectors are used as the initial cluster centers, denoted as . ;

[0146] For each feature vector Calculate its distance from each cluster center and assign it to the nearest cluster: ;

[0147] Set this cluster center as the current cluster center;

[0148] For each cluster Recalculate cluster center The average of all eigenvectors within this cluster:

[0149] ;

[0150] The recalculated cluster center is designated as the recalculated cluster center;

[0151] Calculate the Euclidean distance between the current cluster center and the recalculated cluster center; the calculated value represents the change in cluster center. ;

[0152] in, As the current cluster center, To recalculate the cluster center, The dimension of the feature is the number of features in the feature vector of each user in the game server.

[0153] If the change in cluster center is less than a predetermined threshold, the algorithm is considered to have converged and the iteration is stopped.

[0154] If the cluster center change is greater than or equal to a predetermined threshold, then return to the previous step for each feature vector. Calculate its distance from each cluster center and assign it to the nearest cluster, repeating the steps;

[0155] For each behavioral pattern The analysis uncovers the logic and rules behind user behavior, specifically:

[0156] From clustering results Extracting user behavior feature vectors ;

[0157] Prepare labels for each feature vector. Tags can be user behavior categories, game results, etc.

[0158] use and Models are constructed using algorithms such as decision trees, random forests, and logistic regression. Such as classification models or regression models;

[0159] Extract rules from the constructed model that can explain the logic behind user behavior; for example, the branches of a decision tree can represent different behavioral decision rules.

[0160] Evaluate the model using the validation dataset. The performance of the model is evaluated, and the model is optimized based on performance metrics such as accuracy and recall. The validation dataset is a subset of the original dataset used to evaluate the model's performance during training. It is mainly used to adjust the model's hyperparameters, select the best model version, and avoid overfitting. The validation dataset does not participate in the model's training process, so it can provide a relatively objective performance evaluation.

[0161] This embodiment also provides grouping users in the game server and performing corresponding group training, specifically:

[0162] S1. Randomly extract a user from the game server, designate them as the analysis user, and denot them as... ;

[0163] Randomly select another user from the game server and designate them as the comparison user, denoted as _____. The other users are those on the game server excluding the analysis users;

[0164] S2, Obtain and analyze users respectively Compared with users The eigenvectors of are denoted as . and ;

[0165] S3, Calculation and Analysis User Compared with users Cosine similarity measures the angular difference between two user behavior feature vectors: ;

[0166] in, For vectors The modulus length represents the user being analyzed. The overall strength of behavioral traits For vectors The modulus length indicates the comparison user The overall strength of behavioral characteristics;

[0167] S4. Repeat steps S1-S3 until all other users in the game server have a cosine similarity value with the analyzed user.

[0168] S5. Repeat steps S1-S4 until all users in the game server have been analyzed as users.

[0169] S6. For each analytics user Treat it as a separate set;

[0170] S7, Extract and analyze users All corresponding comparison users ;

[0171] S8, Extract and analyze users Compared with all users Similarity matrix between :

[0172] ;

[0173] S9. Define a filtering function for each analysis user. Find all users with high similarity to it. : ;

[0174] in, The similarity threshold is a threshold set based on business needs or data analysis results to find users with high similarity.

[0175] like Then determine the user analysis Compared with this user High similarity;

[0176] like Then determine the user analysis Compared with this user Low similarity;

[0177] S10, Analyze the user's judgment Compared with this user All users with high similarity Merge into the same set;

[0178] S11. Repeat steps S1-S10 until the similarity comparison of all users on the game server is completed, forming multiple user sets: ;

[0179] Each set Users in the game share similar ways of thinking and gaming habits;

[0180] S12, for each set Initialize an agent : ;

[0181] in, A policy network is a mapping from a state space to an action space, providing guidance to an agent on how to act in a specific situation. Its goal is to learn the optimal policy, that is, the policy that maximizes cumulative reward in long-term operation. The value function is used to evaluate the quality of an agent in a certain state. It reflects the expected cumulative reward that the agent can obtain in the future if it follows a specific policy, starting from the current state. It helps the agent judge the quality of different states, thereby guiding the learning and optimization of the policy network.

[0182] S13. For each set Collect game behavior data and feedback data from all users within this set to form a training dataset. :

[0183] ;

[0184] in, It is a comprehensive feature, including behavioral features. and feedback features ;

[0185] S14. For each agent Conduct independent training;

[0186] S15. Optimize the objective function using the PPO algorithm to maximize the agent's performance in this user set environment: ;

[0187] in, These are the parameters of the policy network, defining the probability distribution of the agent's action selection under different states. Is the policy network in state Select action The probability represents the agent's preference for actions under the current policy. Under the old strategy, i.e., the strategy before the last update, select the action. The probability is used to compare changes between the old and new strategies. It is the dominance function, representing the state... Select action Advantages, measuring the choice of action Compared to the average action in the state The additional rewards that can be obtained guide the agent to choose more advantageous actions. It is the shearing factor, used to control the maximum allowable change between the old and new policies during PPO updates. It limits the magnitude of policy updates and prevents excessive policy updates from causing training instability. The strategy ratio represents the ratio of the old and new strategies to actions. The preference ratio, For the cut operation, ensure that the policy update does not exceed the preset range;

[0188] S16. Optimize the objective function using the gradient ascent method and update the parameters of the policy network:

[0189] ;

[0190] in, It's the learning rate. It is the gradient of the objective function with respect to the policy network parameters;

[0191] S17. Repeat steps S14-S15 until the agent's performance in the user set environment converges, that is, within a certain number of rounds, the fluctuation of the average reward value of multiple rounds is less than the set threshold, or the change of the policy parameters is less than the set threshold, or the change of the mean square error (MSE) of the value function (i.e. the loss value) is less than the set threshold.

[0192] For each intelligent agent Conduct independent training, specifically:

[0193] Execution environment interaction;

[0194] If the game is declared over, the advantage function is calculated using generalized advantage estimation (GAE). :

[0195] ;

[0196] in, This is a discount factor used to calculate the present value of future rewards. Its value ranges from [0,1). When it's close to 1, the agent focuses more on long-term rewards; when it's close to 0, the agent focuses more on immediate rewards. This is the smoothing parameter for GAE, used to control the smoothness of GAE. Its value ranges from [0,1] and is used to balance bias and variance. The time difference (TD) error represents the TD error at time step t+k, used to calculate the dominance function. , For the state value estimation at time step t+k+1, For the state value estimation at time step t+k, This is an estimate of the advantage function, used to measure performance in state... Select action The advantage of this, relative to the value of average movement;

[0197] Update the policy network by maximizing the objective function of PPO:

[0198] ;

[0199] in, This is the shearing factor for PPO, used to control the maximum magnitude of policy updates and prevent excessively large updates from causing training instability. These are the parameters of the policy network, used to define the policy network. The agent's policy is improved by optimizing and adjusting these parameters. In the state Below, policy network Select Action The probability represents the agent's preference for actions under the current policy. To select an action under the old strategy, i.e., the strategy before the last update. The probability is used to compare changes between the old and new strategies, helping to control the magnitude of strategy updates. The strategy ratio represents the ratio of the old and new strategies to actions. The preference ratio, This is a cut operation to ensure that policy updates do not exceed a preset range. This is the advantage function, used to guide the agent to choose a more advantageous action;

[0200] Update the parameters of the policy network using gradient ascent: ;

[0201] in, The learning rate controls the step size for updating the policy network parameters, affecting the convergence speed and stability of training. PPO objective function Policy network parameters The gradient is used to indicate the direction of parameter updates in order to maximize the objective function;

[0202] Update the value function by minimizing the mean squared error: ;

[0203] in, The parameters of the value function network are used to define the value function network. Used to estimate the value of a state. For the value function network in state The estimated value under the given condition represents the agent's understanding of the state. The value estimate is used to assess the expected return in that state. Let be the estimated value of the value function, representing the estimated value of the value function at time step t, used to update the value function network, specifically as follows: , The reward obtained at time step t. This is a discount factor used to balance the importance of immediate and future rewards. For the state value estimation at time step t+1;

[0204] Update the parameters of the value function using gradient descent: ;

[0205] in, The learning rate is the value function's step size, used to control the update speed of the network parameters and affecting the convergence speed and stability of training. For value function loss For value function network parameters The gradient is used to indicate the direction of parameter updates in order to minimize the loss function;

[0206] Repeatedly execute for each agent The overall steps of independent training are to continue until the agent's performance in the user set environment converges, that is, within a certain number of rounds, the fluctuation of the average reward value over multiple rounds is less than a set threshold, or the change in policy parameters is less than a set threshold, or the change in the mean squared error (MSE) of the value function (i.e., the loss value) is less than a set threshold.

[0207] This embodiment also provides cross-set integration training based on the group training situation, specifically as follows:

[0208] Get each collection Corresponding training dataset This will be integrated into a comprehensive dataset.

[0209] ;

[0210] in, It contains the comprehensive feature vectors from all user sets. ;

[0211] Initialize a global agent Training is conducted based on comprehensive characteristics of the entire user group:

[0212] ;

[0213] in, A policy network is a mapping from a state space to an action space, providing guidance to an agent on how to act in a specific situation. Its goal is to learn the optimal policy, that is, the policy that maximizes cumulative reward in long-term operation. The value function is used to evaluate the quality of an agent in a certain state. It reflects the expected cumulative reward that the agent can obtain in the future if it follows a specific policy, starting from the current state. It helps the agent judge the quality of different states, thereby guiding the learning and optimization of the policy network.

[0214] Execution environment interaction;

[0215] If the game is declared over, the advantage function is calculated using GAE. : ;

[0216] in, This is a discount factor used to calculate the present value of future rewards. Its value ranges from [0,1). When it's close to 1, the agent focuses more on long-term rewards; when it's close to 0, the agent focuses more on immediate rewards. This is the smoothing parameter for GAE, used to control the smoothness of GAE. Its value ranges from [0,1] and is used to balance bias and variance. The time difference (TD) error represents the TD error at time step t+k, used to calculate the dominance function. , For the state value estimation at time step t+k+1, For the state value estimation at time step t+k, This is an estimate of the advantage function, used to measure performance in state... Select action The advantage of this, relative to the value of average movement;

[0217] Update the policy network by maximizing the objective function of PPO:

[0218] ;

[0219] in, This is the shearing factor for PPO, used to control the maximum magnitude of policy updates and prevent excessively large updates from causing training instability. These are the parameters of the policy network, used to define the policy network. The agent's policy is improved by optimizing and adjusting these parameters. In the state Below, policy network Select Action The probability represents the agent's preference for actions under the current policy. To select an action under the old strategy, i.e., the strategy before the last update. The probability is used to compare changes between the old and new strategies, helping to control the magnitude of strategy updates. The strategy ratio represents the ratio of the old and new strategies to actions. The preference ratio, This is a cut operation to ensure that policy updates do not exceed a preset range. This is the advantage function, used to guide the agent to choose a more advantageous action;

[0220] Update the parameters of the policy network using gradient ascent: ;

[0221] in, The learning rate controls the step size for updating the policy network parameters, affecting the convergence speed and stability of training. PPO objective function Policy network parameters The gradient is used to indicate the direction of parameter updates in order to maximize the objective function;

[0222] Update the value function by minimizing the mean squared error: ;

[0223] in, The parameters of the value function network are used to define the value function network. Used to estimate the value of a state. For the value function network in state The estimated value under the given condition represents the agent's understanding of the state. The value estimate is used to assess the expected return in that state. Let be the estimated value of the value function, representing the estimated value of the value function at time step t, used to update the value function network, specifically as follows: , The reward obtained at time step t. This is a discount factor used to balance the importance of immediate and future rewards. For the state value estimation at time step t+1;

[0224] Update the parameters of the value function using gradient descent: ;

[0225] in, The learning rate is the value function's step size, used to control the update speed of the network parameters and affecting the convergence speed and stability of training. For value function loss For value function network parameters The gradient is used to indicate the direction of parameter updates in order to minimize the loss function;

[0226] Repeat the overall steps from environmental interaction to updating the parameters of the value function using gradient descent until the agent's performance in the user set environment converges, i.e., within a certain number of rounds, the fluctuation of the average reward value over multiple rounds is less than a set threshold, or the change in policy parameters is less than a set threshold, or the change in the mean squared error (MSE) of the value function (i.e., the loss value) is less than a set threshold.

[0227] The execution environment interaction specifically includes:

[0228] Reset the game environment to get the initial state. ;

[0229] intelligent agent Based on the current policy network In state Select action :

[0230] ;

[0231] in, The current state represents the agent's current state in the environment, used for decision-making and action selection. For the current action, Indicates the agent's state The action selected below is the policy network's action in the state. Select action The probability is used to define the agent's behavioral policy for selecting actions;

[0232] Perform actions in the environment To obtain the next state and rewards ;

[0233] in, The reward value represents the reward the agent receives for performing an action. The instant reward obtained afterward;

[0234] trajectory data Store in buffer ;

[0235] in, The next state indicates that an action will be performed. The new state after the environment shifts, It serves as a buffer to store trajectory data, saving the trajectory data of the agent's interaction with the environment for subsequent policy updates;

[0236] Define the conditions for ending the game, such as the player reaching a specific game objective (e.g., scoring more than a certain value), the player failing (e.g., running out of health points or time), or reaching a preset game duration or number of rounds.

[0237] After each game step, check if the current game state meets any termination conditions;

[0238] If the termination condition is met, the game is considered over.

[0239] If the termination condition is not met, the game is determined not to have ended, and the status is returned. Then repeat the above steps of environmental interaction.

[0240] This embodiment also provides the following: collecting user feedback on the trained agent and forming feedback-driven iterative training based on the user feedback, specifically:

[0241] Select a group of users from all users on the game server. Conduct experience testing:

[0242] ;

[0243] in, It is a collection of all users. It is the set of users selected for experience testing. It is a function that selects a group of users from all users on the game server according to predetermined conditions, such as selecting the most active users from all users on the game server according to a set selection number, or selecting a corresponding number of users in proportion to the set corresponding to each user group according to a set selection number.

[0244] For users Provide a game environment consistent with the training environment, and then train the agent. Deploy to the game environment;

[0245] user With intelligent agents Interaction, generating experience data : ;

[0246] in, To simulate the game interaction process between the user and the global intelligent agent and to evaluate the performance of the intelligent agent, a function is developed. Through actual game interaction, interaction data between the user and the intelligent agent is collected to provide raw materials for subsequent analysis of the performance of the intelligent agent and user satisfaction.

[0247] For users Each user in Collect their direct feedback :

[0248] ;

[0249] in, A function to obtain direct user feedback on the performance of the intelligent agent, so as to directly understand the user's needs and dissatisfactions; used to collect feedback information directly from users, so as to quickly identify the strengths and weaknesses of the intelligent agent and make targeted improvements.

[0250] Collect users The indirect feedback data is then normalized.

[0251] ; ;

[0252] Integrating indirect feedback data : ;

[0253] Direct feedback and indirect feedback Concatenate into a comprehensive feedback vector : ;

[0254] Forming a comprehensive feedback dataset : ;

[0255] For users Each user in Comments Sentiment analysis is performed to identify positive and negative feedback and obtain a sentiment score. : ;

[0256] in, To analyze the sentiment of user reviews and quantify the user's satisfaction or dissatisfaction with the performance of the agent, this function helps development teams quickly understand the sentiment in a large number of user reviews, thereby identifying potential problems in the agent design and providing direction for further optimization.

[0257] Calculate each behavioral feature Importance score : ;

[0258] in, To assess the importance of different features in user feedback and help the team identify which features have the greatest impact on user experience, thus prioritizing the optimization of these features, this function quantifies the importance of each feature to help the development team prioritize improvements, focus resources on optimizing the features that have the greatest impact on user experience, and improve the overall performance of the agent and user satisfaction.

[0259] For the comprehensive feedback dataset Perform clustering to obtain clustering results :

[0260] ;

[0261] Based on the feedback analysis results, define improvement goals. : ;

[0262] Each improvement point This indicates the shortcomings of the intelligent agent in a specific scenario or behavioral pattern;

[0263] Select data relevant to the improvement goals from the original training data. :

[0264] ;

[0265] in, To judge the data Is it in line with the improvement goals? Related functions are used to filter out data that is useful for improvement goals;

[0266] Using the improved dataset Targeted training of the agent and updating of the policy network and value function : ;

[0267] in, This is a function for training using the PPO algorithm, used to train and return an improved policy network and value function based on provided data and an initial model;

[0268] Update the improved policy network and value function to the global agent:

[0269] ; ;

[0270] Repeated execution for users Each user in Collect their direct feedback The overall steps from updating the improved policy network and value function to the global agent are continued until the agent's performance reaches a satisfactory level. This involves continuously training and optimizing the agent until its performance meets predetermined performance standards or goals. Specifically, this could mean that the agent's cumulative reward reaches a predetermined threshold (e.g., the agent's score in a game remains stable at a high level), the agent's win rate in competitive games reaches a satisfactory level (e.g., a win rate of over 70%), the agent's performance remains stable over multiple evaluation periods without large fluctuations, the agent's policy parameters change very little, and the change in the loss function (i.e., the loss value) during training is less than a set threshold.

[0271] Example 3 is an improvement upon Example 2. In this example, other games related to the current game are identified, and data is collected and analyzed. Specifically:

[0272] Analyze the current game's genre, gameplay, and user base to identify other games related to it: ;

[0273] in, A function to identify other games related to the current game is used to find other games with similar characteristics by analyzing features such as the current game's type, gameplay, and user base. This expands the data sources, provides more diverse data for model training, and improves the agent's generalization ability. This is the current game. It is the identified set of related games;

[0274] For each relevant game To obtain user data separately: ;

[0275] in, From related games User data obtained from To the specified related games Functions for collecting user data in the game are used to obtain user data in the relevant game, including but not limited to user game behavior, operation records, decision choices, etc., to enrich the user behavior dataset and help the agent better understand user behavior patterns in different game environments;

[0276] For each relevant game Data is collected and user behavior is analyzed for each user in the process, resulting in... and That is, for each relevant game Each user in the process performs the steps of claims 2 and 3;

[0277] The user data from related games is integrated into the user data of the current game to form the final training dataset: ;

[0278] in, It is the final training dataset after expansion.

[0279] This embodiment also provides that all training data from the current game and related games are fused and finally trained to generate a general intelligent agent that can adapt to various user behaviors and habits, specifically:

[0280] Obtain the final training dataset ;

[0281] For the final training dataset Perform cross-game user grouping and training, that is, execute the contents of claims 4-6 on the final training dataset, specifically:

[0282] Compare users of other games with users of the current game, and group users with high similarity into the same set;

[0283] Joint training is performed on data from each set (containing users of the current game and related games) to enable the agent to adapt to a wider range of user behavior characteristics.

[0284] Example 4: This example also discloses a system for implementing a game agent design method based on deep reinforcement learning. (See attached document.) Figure 2 ,include:

[0285] Data acquisition module: Collects game data of users in the current game and related games, that is, the user's game playing situation, and collects the user's feedback on the game and the trained agent;

[0286] Data Analysis Module: Analyzes game data of users of the current game and related games, classifies users, and groups users with similar game operation strategies into the same set for data training;

[0287] Data training module: Based on the user classification set of the current game or related games, first train the agent on the data within each set, and then train the agent as a whole according to the training status of the sets;

[0288] Data integration module: Integrates the training data of the agent for the current game and the training data of the agent for related games, and then trains the agent again;

[0289] Data generation module: Generates or updates the agent based on the training results.

Claims

1. A method for designing game intelligent agents based on deep reinforcement learning, characterized in that: include: Data is collected from all users on the game server. Perform user behavior analysis on all users on the game server; Users on the game server are grouped and trained accordingly. Based on the results of group training, perform cross-set integration training; Collect user feedback on the trained agent and form feedback-driven iterative training based on the user feedback; Identify other games related to the current game and collect and analyze the data; Based on data collected and analyzed from other games related to the current game, cross-game user grouping and training are conducted; By fusing all training data from the current game and related games and performing final training, a general intelligent agent capable of adapting to various user behaviors and habits is generated. Collect user feedback on the trained agent and use this feedback to generate feedback-driven iterative training, specifically: Select a group of users from all users on the game server. Conduct experience testing: ； in, It is a collection of all users. It is the set of users selected for experience testing. It is a function that selects a group of users from all users on the game server based on predetermined conditions; For users Provide a game environment consistent with the training environment, and then train the agent. Deploy to the game environment; user With intelligent agents Interaction, generating experience data : ; in, To simulate the game interaction process between the user and the global intelligent agent and to evaluate the performance of the intelligent agent, a function is developed. Through actual game interaction, interaction data between the user and the intelligent agent is collected to provide raw materials for subsequent analysis of the performance of the intelligent agent and user satisfaction. For users Each user in Collect their direct feedback : ； in, To obtain direct user feedback on the performance of the intelligent agent; Collect users The indirect feedback data is then normalized. ； ； in, The normalized game duration The retention rate after normalization. and These are the maximum and minimum game times for all users, respectively. and These are the maximum and minimum retention rates for all users, respectively. and These are the game duration and retention rate before normalization; Integrating indirect feedback data : ; Direct feedback and indirect feedback Concatenate into a comprehensive feedback vector : ; Forming a comprehensive feedback dataset : ; For users Each user in Comments Sentiment analysis is performed to identify positive and negative feedback and obtain a sentiment score. : ; in, A function for analyzing the sentiment of user comments; Calculate each behavioral feature Importance score : ; in, To assess the importance of different features in user feedback and help the team identify which features have the greatest impact on user experience, functions that optimize these features can be prioritized. For the comprehensive feedback dataset Perform clustering to obtain clustering results : ； Based on the feedback analysis results, define improvement goals. : ; Select data relevant to the improvement goals from the original training data. : ； in, To judge the data Is it in line with the improvement goals? Related functions are used to filter out data that is useful for improvement goals; Using the improved dataset Targeted training of the agent and updating of the policy network and value function : ; in, This is a function for training using the PPO algorithm, used to train and return an improved policy network and value function based on the provided data and initial model; Update the improved policy network and value function to the global agent: ； ； Repeated execution for users Each user in Collect their direct feedback The overall steps from updating the improved policy network and value function to the global agent are continued until the agent's performance reaches a satisfactory level.

2. The method for designing a game agent based on deep reinforcement learning according to claim 1, characterized in that: Data collection is performed on all users on the game server, specifically as follows: For each user Extract its raw behavioral data from the game server, denoted as : ； in, This represents the user's behavior across different dimensions, which are the results of classifying the raw data according to dimensions such as operational, strategic, social, and achievement categories. Using feature extraction functions Transform raw behavioral data into low-dimensional feature vectors : ； in, It was before The weight matrices of the principal components are used to project the original data onto the matrix formed by the previous components. In the low-dimensional subspace defined by each principal component It indicates that it is before The transpose of the weight matrices of the principal components. This represents the standardized raw behavioral data; Collect users Direct feedback data, denoted as : ; The indirect feedback data is normalized to obtain and : ； ； in, The normalized game duration The retention rate after normalization. and These are the maximum and minimum game times for all users, respectively. and These are the maximum and minimum retention rates for all users, respectively. and These are the game duration and retention rate before normalization; Integrating direct feedback and normalized indirect feedback, a comprehensive feedback data is formed, denoted as... : ； behavioral characteristics and feedback features By splicing together, we obtain the comprehensive features. : .

3. The method for designing a game agent based on deep reinforcement learning according to claim 1, characterized in that: User behavior analysis is performed on all users on the game server, specifically as follows: Obtain the feature vector of each user in the game server ; Clustering the behavioral characteristics of all users to identify different behavioral patterns: ; in, It is the set of behavioral feature vectors of all users. It is the clustering result; For each behavioral pattern Analyze and uncover the logic and rules behind user behavior: ； in, It is a behavioral pattern The corresponding set of feature vectors, These are the corresponding behavioral pattern tags. It is a behavioral pattern model that is constructed to understand behavioral patterns. The logic and rules governing user behavior.

4. The method for designing a game agent based on deep reinforcement learning according to claim 1, characterized in that: Users on the game server are grouped and trained accordingly, specifically as follows: S1. Randomly extract a user from the game server, designate them as the analysis user, and denot them as... ; Randomly select another user from the game server and designate them as the comparison user, denoted as _____. ; S2, Obtain and analyze users respectively Compared with users The eigenvectors of are denoted as . and ; S3, Calculation and Analysis User Compared with users Cosine similarity: ； in, For vectors The modulus length represents the user analysis. The overall strength of behavioral traits For vectors The modulus length indicates the comparison user The overall strength of behavioral characteristics; S4. Repeat steps S1-S3 until all other users in the game server have a cosine similarity value with the analyzed user. S5. Repeat steps S1-S4 until all users in the game server have been analyzed as users. S6. For each analytics user Treat it as a separate set; S7, Extract and analyze users All corresponding comparison users ; S8, Extract and analyze users Compared with all users Similarity matrix between : ； S9. Define a filtering function for each analysis user. Find all users with high similarity to it. : ； in, The similarity threshold; like Then determine the user analysis Compared with this user High similarity; like Then determine the user analysis Compared with this user Low similarity; S10, Analyze the user's judgment Compared with this user All users with high similarity Merge into the same set; S11. Repeat steps S1-S10 until the similarity comparison of all users on the game server is completed, forming multiple user sets: ; Each set Users in the game share similar ways of thinking and gaming habits; S12, for each set Initialize an agent : ; in, A policy network is a mapping from a state space to an action space, providing guidance to an agent on how to act in a specific situation. Its goal is to learn the optimal policy, that is, the policy that maximizes cumulative reward in long-term operation. The value function is used to evaluate the quality of an agent in a certain state. It reflects the expected cumulative reward that the agent can obtain in the future if it follows a specific policy, starting from the current state. It helps the agent judge the quality of different states, thereby guiding the learning and optimization of the policy network. S13. For each set Collect game behavior data and feedback data from all users within this set to form a training dataset. : ; in, It is a comprehensive feature, including behavioral features. and feedback features ; S14. For each agent Conduct independent training; S15. Maximize the agent's performance in this user set environment: ； in, These are the parameters of the policy network, defining the probability distribution of the agent's action selection under different states. Is the policy network in state Select action The probability represents the agent's preference for actions under the current policy. Under the old strategy, i.e., the strategy before the last update, select the action. The probability is used to compare changes between the old and new strategies. It is the dominance function, representing the state... Select action Advantages, measuring the choice of action Compared to the average action in the state The additional rewards that can be obtained guide the agent to choose more advantageous actions. It is the shearing factor, used to control the maximum allowable change between the old and new policies during PPO updates. It limits the magnitude of policy updates and prevents excessive policy updates from causing training instability. The strategy ratio represents the ratio of the old and new strategies to actions. The preference ratio, For the cut operation, ensure that the policy update does not exceed the preset range; S16. Optimize the objective function and update the parameters of the policy network: ; in, It's the learning rate. It is the gradient of the objective function with respect to the policy network parameters; S17. Repeat steps S14-S15 until the agent's performance in the user set environment converges.

5. The method for designing a game agent based on deep reinforcement learning according to claim 1, characterized in that: Based on the group training results, cross-set integration training is performed, specifically as follows: Get each collection Corresponding training dataset This will be integrated into a comprehensive dataset. ； Initialize a global agent : ; in, A policy network is a mapping from a state space to an action space, providing guidance to an agent on how to act in a specific situation. Its goal is to learn the optimal policy, that is, the policy that maximizes cumulative reward in long-term operation. The value function is used to evaluate the quality of an agent in a certain state. It reflects the expected cumulative reward that the agent can obtain in the future if it follows a specific policy, starting from the current state. It helps the agent judge the quality of different states, thereby guiding the learning and optimization of the policy network. Execution environment interaction; After the game ends, use GAE to calculate the advantage function. : ; in, It is a discount factor used to calculate the present value of future rewards. This is the smoothing parameter of GAE, used to control the smoothness of GAE and balance bias and variance. The time difference (TD) error represents the TD error at time step t+k and is used to calculate the dominance function. Update the policy network by maximizing the objective function of PPO: ； in, This is the shearing factor for PPO, used to control the maximum magnitude of policy updates and prevent excessively large updates from causing training instability. These are the parameters of the policy network, used to define the policy network. By optimizing and adjusting these parameters, the agent's policy can be improved. In the state Below, policy network Select Action The probability represents the agent's preference for actions under the current policy. To select an action under the old strategy, i.e., the strategy before the last update. The probability is used to compare changes between the old and new strategies, helping to control the magnitude of strategy updates. The strategy ratio represents the ratio of the old and new strategies to actions. The preference ratio, This is a cut operation to ensure that policy updates do not exceed a preset range. This is the advantage function, used to guide the agent to choose a more advantageous action; Update the parameters of the policy network using gradient ascent: ; in, The learning rate controls the step size for updating the policy network parameters, affecting the convergence speed and stability of training. PPO objective function Policy network parameters The gradient is used to indicate the direction of parameter updates in order to maximize the objective function; Update the value function by minimizing the mean squared error: ; in, The parameters of the value function network are used to define the value function network. Used to estimate the value of a state. For the value function network in state The estimated value under the given condition represents the agent's understanding of the state. The value estimate is used to assess the expected return in that state. Let be the estimated value of the value function, representing the estimated value of the value function at time step t, used to update the value function network, specifically as follows: , The reward obtained at time step t. This is a discount factor used to balance the importance of immediate and future rewards. For the state value estimation at time step t+1; Update the parameters of the value function using gradient descent: ; in, The learning rate is the value function's step size, used to control the update speed of the network parameters and affecting the convergence speed and stability of training. For value function loss For value function network parameters The gradient is used to indicate the direction of parameter updates in order to minimize the loss function.

6. The method for designing a game agent based on deep reinforcement learning according to claim 1, characterized in that: Identify other games related to the current game and collect and analyze the data, specifically: Analyze the current game's genre, gameplay, and user base to identify other games related to it: ; in, To identify functions of other games related to the current game, This is the current game. It is the identified set of related games; For each relevant game To obtain user data separately: ; in, From related games User data obtained from To the specified related games Functions for collecting user data; For each relevant game Data is collected and user behavior is analyzed for each user in the process, resulting in... and ; The user data from related games is integrated into the user data of the current game to form the final training dataset: ; in, It is the final training dataset after expansion.

7. The method for designing a game agent based on deep reinforcement learning according to claim 1, characterized in that: The training data from the current game and related games are merged and then trained to generate a general-purpose intelligent agent capable of adapting to various user behaviors and habits. Specifically: Obtain the final training dataset ; For the final training dataset Conduct cross-game user grouping and training.

8. A system for implementing the game agent design method based on deep reinforcement learning as described in claim 1, characterized in that: include: Data acquisition module: Collects game data from users of the current game and related games, and collects user feedback on the game and the trained agent; Data Analysis Module: Analyzes game data of users of the current game and related games, classifies users, and groups users with similar game operation strategies into the same set for data training; Data training module: Based on the user classification set of the current game or related games, first train the agent on the data within each set, and then train the agent as a whole according to the training status of the sets; Data integration module: Integrates the training data of the agent for the current game and the training data of the agent for related games, and then trains the agent again; Data generation module: Generates or updates the agent based on the training results.

Images