Game adaptive difficulty adjustment method and system

Abstract

The invention relates to the technical field of game control, in particular to a game self-adaption difficulty adjusting method and system.The game self-adaption difficulty adjusting method comprises the steps that an operation sequence and game state data of a player in a target game are obtained, and feature fusion is conducted on the operation sequence and the game state data of the player to obtain comprehensive ability features of the player; and carrying out hidden space modeling on the comprehensive capability characteristics of the player according to a variational auto-encoder to obtain a player hidden space variable, and carrying out t-SNE dimensionality reduction on the player hidden space variable. According to the method, by obtaining the operation sequence and the game state data of the players and combining the variational auto-encoder and t-SNE dimensionality reduction, the comprehensive ability characteristics of each player can be accurately modeled, the difficulty of the game can be dynamically adjusted according to the ability characteristics of the players, the game is made to adapt to the levels of different players, and therefore the game quality is improved. A player does not feel too difficult and too simple in the game process, so that challenging and interestingness of the game are kept.

Description

Technical Field

[0001] The present invention relates to the field of game control technology, and in particular to a game adaptive difficulty adjustment method and system. Background Art

[0002] Traditional methods usually rely on game developers to manually set a fixed difficulty level, or use predetermined rules to adjust the difficulty. This method lacks personalization and cannot dynamically adjust the difficulty according to the actual performance and ability characteristics of each player, which can easily lead to some players experiencing a situation that is too simple or too difficult. Players may lose interest due to the mismatch of game difficulty, affecting the challenge and fun of the game; and traditional methods often cannot flexibly adjust the difficulty of the game according to the real-time performance and feedback of the players. The difficulty is usually pre-set and cannot quickly respond to the changes in the player's operation in the game, which means that players may not get timely difficulty adjustments when facing challenges, resulting in an unsmooth gaming experience, or even causing the game to be too boring or too frustrating; and traditional methods lack the ability of deep learning and intelligent optimization. In many traditional games, the difficulty adjustment strategy is static, and the difficulty level and task design in the game are mostly manually set based on the developer's experience; and because traditional methods often rely on manually designed difficulty rules, game developers need to invest a lot of time and energy in testing and adjustment to ensure that all difficulty settings can adapt to the needs of different players, and each adjustment requires manual modification, and the test cycle during update is long. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a game adaptive difficulty adjustment method and system.

[0004] The technical solution adopted to solve the above technical problems is: a method for adjusting game adaptive difficulty, comprising:

[0005] Acquire the operation sequence and game status data of the player in the target game, and perform feature fusion on the operation sequence and game status data of the player to obtain the comprehensive ability characteristics of the player;

[0006] Performing latent space modeling on the comprehensive ability characteristics of the player according to the variational autoencoder to obtain the player latent space variables, and performing t-SNE dimension reduction on the player latent space variables to obtain the reduced dimension player latent space variables;

[0007] Performing category clustering on the reduced-dimensional player latent space variables according to the K-means algorithm to obtain category labels of the players;

[0008] Constructing a difficulty adjustment agent according to the reduced-dimensional player latent space variables, the player's category labels, and the historical difficulty sequence;

[0009] The difficulty adjustment agent is adjusted according to the deep deterministic policy gradient to obtain a difficulty adjustment strategy for the target game.

[0010] Preferably, the operation sequence includes key frequency, reaction time, movement path complexity and combat decision entropy, and the game status data includes level passing time, resource consumption rate and number of failures.

[0011] Preferably, obtaining the operation sequence and game status data of the player in the target game includes:

[0012] The key input frequency of the player in the time window is recorded by calculating the number of keys in the preset time window, that is, obtaining the key frequency;

[0013] Record the time difference from when the player starts to observe the target game state change to when the player performs an operation, that is, obtain the reaction time;

[0014] Record all positions of the player in the target game to obtain a coordinate point sequence, and calculate the change of position at each moment according to the coordinate point sequence to obtain a differential time series, wherein the expression of the differential time series is as follows:

[0015] X(t) = U(t) - U(t-1);

[0016] Among them, X(t) represents the differential time series at time t, and U(t) represents the coordinate point sequence at time t;

[0017] The cumulative deviation is calculated according to the differential time series, and the fluctuation range and standard deviation are calculated according to the cumulative deviation, wherein the calculation formula of the cumulative deviation is as follows:

[0018]

[0019] Where Y(t) represents the cumulative deviation at time t;

[0020] The calculation formula of the fluctuation range is as follows:

[0021] R(t)=max(Y(t))-min(Y(t));

[0022] Among them, R(t) represents the fluctuation range at time t;

[0023] The moving path complexity is calculated according to the fluctuation range and the standard deviation, wherein the calculation formula of the moving path complexity is as follows:

[0024]

[0025] Among them, Cp represents the complexity of the moving path, and S(t) represents the standard deviation at time t.

[0026] Preferably, obtaining the operation sequence and game status data of the player in the target game also includes:

[0027] Record each decision action of the player and calculate the selection probability of the decision action, wherein the calculation formula of the selection probability is as follows:

[0028]

[0029] Among them, P(A i ) represents decision action A i The probability of selection, n i Represents decision action A i The number of times selected, T represents the total number of action decisions;

[0030] The combat decision entropy of the player is calculated according to the selection probability of the decision action, wherein the calculation formula of the combat decision entropy is as follows:

[0031]

[0032] Among them, H d represents the player's combat decision entropy, and N represents the total number of decision actions;

[0033] Recording the time taken by the player to pass each level in the target game, that is, obtaining the level passing time;

[0034] Recording the resource consumption rate of the player during the target game, that is, obtaining the resource consumption rate;

[0035] The number of times the player fails in the target game is recorded, that is, the number of failures is obtained.

[0036] Preferably, the operation sequence and game state data of the player are subjected to feature fusion to obtain the comprehensive ability features of the player, including:

[0037] Merging the player's operation sequence and game state data to obtain a merged feature vector, and normalizing the merged feature vector to obtain a normalized merged feature vector;

[0038] The information entropy of each feature in the normalized merged feature vector is calculated, and the weight of each feature in the normalized merged feature vector is calculated according to the information entropy of the feature, wherein the calculation formula of the weight of the feature is as follows:

[0039]

[0040] Among them, ω i represents the weight of the i-th feature in the normalized merged feature vector, α represents the hyperparameter that controls the steepness of the weight distribution, and I(x i ) represents the information entropy of the i-th feature in the normalized merged feature vector;

[0041] The comprehensive ability feature of the player is determined according to the weight of each feature in the normalized combined feature vector and the normalized combined feature vector.

[0042] Preferably, the comprehensive ability characteristics of the player are modeled in latent space according to a variational autoencoder to obtain the player latent space variables, including:

[0043] The mean and standard deviation of the potential distribution of the comprehensive ability characteristics of the player are learned according to the variational autoencoder, wherein the calculation formula of the mean is as follows:

[0044] μ z =W μ ReLU(W e F p +b e );

[0045] Among them, μ z represents the mean of the potential distribution of the player’s comprehensive ability characteristics, W μ represents the trainable weight matrix used to generate the mean, W e represents a trainable weight matrix, ReLU represents the activation function, and F p Indicates the comprehensive ability characteristics of the player, b e represents the bias term;

[0046] The calculation formula of the mean is as follows:

[0047]

[0048] Among them, σ z represents the standard deviation of the potential distribution of the player’s comprehensive ability characteristics, W σ represents the trainable weight matrix used to generate the standard deviation;

[0049] The mean and standard deviation of the potential distribution of the comprehensive ability characteristics of the player are sampled according to the reparameterization to obtain the player latent space variable, wherein the calculation formula of the player latent space variable is as follows:

[0050] z=μ z +σ z ☉∈,∈~N(0,I);

[0051] Among them, z represents the player latent space variable, ∈ represents the noise sampled from the standard normal distribution, and ⊙ represents element-wise multiplication.

[0052] Preferably, the reduced-dimensional player latent space variables are clustered according to a K-means algorithm to obtain a category label of the player, including:

[0053] Construct an objective function, which is used to minimize the sum of squares of distances from points within the cluster to the center of the cluster, where the objective function is as follows:

[0054]

[0055] Among them, J represents the sum of squares of distances from points within the cluster to the cluster center, and p i represents the latent space variable of the reduced-dimensional player, r ik Represents the dimension-reduced player latent space variable p i Whether it belongs to cluster k, o k represents the center of cluster k, ||p i -o k || 2 Represents the dimension-reduced player latent space variable x i The square of the distance to the center of cluster k;

[0056] Allocating the reduced-dimensional player latent space variables corresponding to each of the players to different clusters based on the objective function;

[0057] The silhouette coefficient of the cluster is calculated, and the silhouette coefficient of the cluster is compared with a preset silhouette coefficient threshold. If the silhouette coefficient of the cluster is greater than the preset silhouette coefficient threshold, it indicates that the reduced-dimensional player latent space variables corresponding to each of the players are clustered. Otherwise, it indicates that the reduced-dimensional player latent space variables corresponding to each of the players are not clustered. The calculation formula of the silhouette coefficient is as follows:

[0058]

[0059] Among them, S(i) represents the silhouette coefficient of the cluster, a(i) represents the average distance from the reduced-dimensional player latent space variable to other points in its cluster, and b(i) represents the average distance from the reduced-dimensional player latent space variable to the nearest other clusters.

[0060] Preferably, constructing a difficulty adjustment agent according to the dimension-reduced player latent space variables, the player's category labels and the historical difficulty sequence comprises:

[0061] The state of the difficulty adjustment agent is constructed according to the dimension-reduced player latent space variables, the player's category label and the historical difficulty sequence, wherein the expression of the state is as follows:

[0062] s t =(zt ,c p ,{d t-τ ,...,d t-1});

[0063] Among them, s t represents the state of the difficulty adjustment agent, c p Indicates the player's category label, {d t-τ ,...,d t-1} represents the historical difficulty sequence;

[0064] The action of the difficulty adjustment agent is constructed according to the preset difficulty parameter, wherein the expression of the action is as follows:

[0065] a t =[δ enemy ,δ resource , δ puzzle ];

[0066] Among them, a t represents the action of the difficulty adjustment agent, δ enemy Indicates the strength of controlling the enemy AI, δ resource Indicates the richness of control resources, δ puzzle It controls the difficulty of solving puzzles in the game;

[0067] The reward of the difficulty adjustment agent is constructed according to the number of successes and failures of the target game of the player, wherein the expression of the reward is as follows:

[0068] r t =0.7r immersion +0.3r frustration ;

[0069] Among them, r t represents the reward of the difficulty-adjusted agent, r immersion represents the immersion level, and r immersion =tanh(2C f -0.5), C f represents the challenge balance coefficient, and tanh represents the hyperbolic tangent function, N success Represents the number of successes of the player in the time window, N attempts Indicates the total number of attempts, N fail represents the number of failures of the player in the time window, r frustration represents the degree of frustration, and r frustration = -log(N fail ).

[0070] Preferably, adjusting the difficulty adjustment agent according to the deep deterministic policy gradient to obtain the difficulty adjustment strategy of the target game includes:

[0071] Uniformly random selection of actions with a preset probability;

[0072] Execute the action and obtain the next state and reward at the same time;

[0073] Put the current strategy into the experience playback buffer, randomly and evenly collect multiple strategies, and update the main network parameters for each strategy according to the loss function, where the trajectory includes the current state, current action, next moment state and reward;

[0074] The target network parameters are updated using the main network parameters every preset number of training steps.

[0075] The technical solution adopted to solve the above technical problem is: a game adaptive difficulty adjustment system, which is applicable to the game adaptive difficulty adjustment method, including:

[0076] A capability characterization unit, the capability characterization unit is used to obtain the operation sequence and game status data of the player in the target game, and perform feature fusion on the operation sequence and game status data of the player to obtain the comprehensive capability characteristics of the player;

[0077] A spatial modeling unit, the spatial modeling unit is used to perform latent space modeling on the comprehensive ability characteristics of the player according to the variational autoencoder to obtain the player latent space variables, and perform t-SNE dimension reduction on the player latent space variables to obtain the reduced dimension player latent space variables;

[0078] A player classification unit, the player classification unit is used to classify the reduced-dimensional player latent space variables according to a K-means algorithm to obtain a category label of the player;

[0079] An agent construction unit, the agent construction unit is used to construct a difficulty adjustment agent according to the dimension reduction player latent space variables, the player's category label and the historical difficulty sequence;

[0080] A difficulty adjustment unit, wherein the difficulty adjustment unit is used to adjust the difficulty adjustment agent according to the deep deterministic policy gradient to obtain a difficulty adjustment strategy for the target game.

[0081] The beneficial effects of the present invention are as follows: (1) The present invention can accurately model the comprehensive ability characteristics of each player by acquiring the player's operation sequence and game status data, combining variational autoencoders and t-SNE dimensionality reduction. According to the player's ability characteristics, the method can dynamically adjust the difficulty of the game to adapt it to the level of different players. In this way, the player will neither feel too difficult nor too simple during the game, thereby maintaining the challenge and fun of the game. The difficulty adjustment agent is adjusted through the deep deterministic policy gradient (DDPG) algorithm. The method can automatically optimize the difficulty adjustment strategy, which means that the game can automatically adapt according to the player's real-time feedback and continuously adjust to provide the most appropriate challenge, reducing the need for manual adjustment and improving the game's intelligence. (2) The present invention abstracts and extracts the comprehensive ability characteristics of players through latent space modeling, and t-SNE dimensionality reduction further simplifies these high-dimensional features into low-dimensional space, making the understanding of player abilities more intuitive and accurate. This method effectively reduces the difficulty of analyzing complex player behaviors and can accurately identify the ability differences of players, thereby achieving more refined difficulty adjustment; (3) The present invention clusters the player latent space variables after dimensionality reduction through the K-means algorithm, and adjusts the difficulty based on the historical difficulty sequence, which can help game developers understand the game needs of different player groups. The clustering results can help the game system identify which players prefer high-difficulty or low-difficulty game environments, and better predict the future difficulty needs of players based on historical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A schematic flow chart of the steps of an overall method in an embodiment of the present invention;

[0083] Figure 2 A schematic diagram of the system architecture of an overall system in an embodiment of the present invention.

[0084] Figure numerals: 1. Ability characterization unit; 2. Space modeling unit; 3. Player classification unit; 4. Agent construction unit; 5. Difficulty adjustment unit. DETAILED DESCRIPTION

[0085] Embodiment 1, as Figure 1 As shown, a method for adaptively adjusting game difficulty proposed by the present invention comprises:

[0086] S1. Obtain the operation sequence and game status data of the player in the target game, and perform feature fusion on the operation sequence and game status data of the player to obtain the comprehensive ability characteristics of the player;

[0087] S2. Perform latent space modeling on the comprehensive ability characteristics of the player according to the variational autoencoder to obtain the player latent space variables, and perform t-SNE dimension reduction on the player latent space variables to obtain the reduced dimension player latent space variables;

[0088] S3, classifying the reduced-dimensional player latent space variables according to the K-means algorithm to obtain the player's category label;

[0089] S4, constructing a difficulty adjustment agent based on the reduced-dimensionality player latent space variables, the player's category labels, and the historical difficulty sequence;

[0090] S5. Adjust the difficulty adjustment agent according to the deep deterministic policy gradient to obtain the difficulty adjustment strategy of the target game.

[0091] In the present invention, the autoencoder is an unsupervised learning model for learning low-dimensional representations of data. The variational autoencoder is a variant of the autoencoder. By introducing the variational inference technology, the data is mapped to the latent space and new data can be generated. Latent space modeling refers to finding a low-dimensional implicit representation (i.e., latent space variables) by modeling the player's operation and game state characteristics, so that the player's ability characteristics can be more efficiently represented. t-SNE is a nonlinear dimensionality reduction technology for mapping high-dimensional data to a low-dimensional space, usually two-dimensional or three-dimensional, for data visualization. It can maintain the distance between similar data points in the data and help understand the structure of the data. K-means is a common clustering algorithm. By dividing the data into pre-set K clusters and finding the center (mean) of each cluster, the data points in each cluster are made as similar as possible. It is used here to cluster the latent space features of the players to distinguish players of different categories. In the clustering process, each player will be Assign a category label, which indicates which category the player belongs to. The player's category label can be used to help the game system adjust the game difficulty according to different types of players; the difficulty adjustment agent is an artificial intelligence algorithm that is responsible for dynamically adjusting the difficulty of the game based on the player's ability, category label and historical difficulty information. Its purpose is to ensure that the player receives appropriate challenges during the game; Deep Deterministic Policy Gradient (DDPG) is an algorithm based on deep reinforcement learning, which is particularly suitable for tasks in continuous action space. In the context of game difficulty adjustment, DDPG dynamically adjusts the game difficulty by learning the optimal strategy, so that the player is always in a balance between challenge and entertainment; the difficulty adjustment strategy is the final strategy learned by the agent, which is used to decide how to change the difficulty of the game in different situations in order to provide players with a suitable gaming experience. For example, when the player behaves very easily, the agent may increase the difficulty of the game; when the player encounters difficulties, the agent may reduce the difficulty.

[0092] Embodiment 2, a method for adaptively adjusting the difficulty of a game proposed by the present invention, compared with embodiment 1, this embodiment also includes: the operation sequence includes key frequency, reaction time, movement path complexity and combat decision entropy, and the game status data includes level passing time, resource consumption rate and number of failures.

[0093] In an optional embodiment, obtaining the operation sequence and game status data of the player in the target game includes:

[0094] A1. Count the number of keys in a preset time window and record the key input frequency of the player in the time window, that is, obtain the key frequency;

[0095] A2. Record the time difference from when the player starts to observe the target game state change to when he performs the operation, that is, the reaction time;

[0096] A3. Record all positions of the player in the target game to obtain a coordinate point sequence. Calculate the change in position at each moment based on the coordinate point sequence to obtain a differential time series. The expression of the differential time series is as follows:

[0097] X(t) = U(t) - U(t-1);

[0098] Among them, X(t) represents the differential time series at time t, and U(t) represents the coordinate point sequence at time t;

[0099] A4. Calculate the cumulative deviation based on the difference time series, and calculate the fluctuation range and standard deviation based on the cumulative deviation. The calculation formula of the cumulative deviation is as follows:

[0100]

[0101] Where Y(t) represents the cumulative deviation at time t;

[0102] The calculation formula of the fluctuation range is as follows:

[0103] R(t)=max(Y(t))-min(Y(t));

[0104] Among them, R(t) represents the fluctuation range at time t;

[0105] A5. Calculate the moving path complexity based on the fluctuation range and standard deviation. The calculation formula of the moving path complexity is as follows:

[0106]

[0107] Among them, C p represents the complexity of the moving path, and S(t) represents the standard deviation at time t.

[0108] It should be noted that the complexity of the player's movement path within a certain period of time, generally the higher the complexity, the more delicate or difficult the player's operation is; decision-making actions refer to every operation choice made by the player during the game. For example, the player's decision to attack, defend or use skills in battle is a decision-making action.

[0109] In an optional embodiment, obtaining the operation sequence and game status data of the player in the target game also includes:

[0110] A6. Record each decision action of the player and calculate the selection probability of the decision action. The calculation formula of the selection probability is as follows:

[0111]

[0112] Among them, P(A i ) represents decision action A i The probability of selection, n i Represents decision action A i The number of times selected, T represents the total number of action decisions;

[0113] A7. Calculate the player's combat decision entropy based on the probability of selecting the decision action. The calculation formula of combat decision entropy is as follows:

[0114]

[0115] Among them, H d represents the player's combat decision entropy, and N represents the total number of decision actions;

[0116] A8. Record the time it takes the player to pass each level in the target game, that is, obtain the level passing time;

[0117] A9. Record the resource consumption rate of the player during the target game, that is, obtain the resource consumption rate;

[0118] A10. Record the number of times the player fails in the target game, that is, obtain the number of failures.

[0119] In an optional embodiment, the player's operation sequence and game state data are subjected to feature fusion to obtain the player's comprehensive ability features, including:

[0120] B1. Merging the player's operation sequence and the game state data to obtain a merged feature vector, and normalizing the merged feature vector to obtain a normalized merged feature vector;

[0121] B2. Calculate the information entropy of each feature in the normalized merged feature vector, and calculate the weight of each feature in the normalized merged feature vector according to the information entropy of the feature, wherein the calculation formula of the feature weight is as follows:

[0122]

[0123] Among them, ω i represents the weight of the i-th feature in the normalized merged feature vector, α represents the hyperparameter that controls the steepness of the weight distribution, and I(x i ) represents the information entropy of the i-th feature in the normalized merged feature vector;

[0124] B3. Determine the comprehensive ability characteristics of the player based on the weight of each feature in the normalized combined feature vector and the normalized combined feature vector.

[0125] In an optional embodiment, the comprehensive ability characteristics of the player are modeled in latent space according to the variational autoencoder to obtain the player latent space variables, including:

[0126] C1. Learn the mean and standard deviation of the potential distribution of the comprehensive ability characteristics of the players based on the variational autoencoder. The calculation formula of the mean is as follows:

[0127] μ z =W μ ReLU(W e F p +b e );

[0128] Among them, μ z represents the mean of the potential distribution of the player’s comprehensive ability characteristics, W μ represents the trainable weight matrix used to generate the mean, W e represents a trainable weight matrix, ReLU represents the activation function, and F p Indicates the comprehensive ability characteristics of the player, b e represents the bias term;

[0129] The calculation formula of the mean is as follows:

[0130]

[0131] Among them, σ z represents the standard deviation of the potential distribution of the player’s comprehensive ability characteristics, W σ represents the trainable weight matrix used to generate the standard deviation;

[0132] C2. Sample the mean and standard deviation of the potential distribution of the player's comprehensive ability characteristics according to the reparameterization to obtain the player's latent space variables, where the calculation formula of the player's latent space variables is as follows:

[0133] z=μ z +σ z ☉∈,∈~N(0,I);

[0134] Among them, z represents the player latent space variable, ∈ represents the noise sampled from the standard normal distribution, and ⊙ represents element-wise multiplication.

[0135] In an optional embodiment, the reduced-dimensional player latent space variables are clustered according to the K-means algorithm to obtain the player's category label, including:

[0136] D1. Construct an objective function, which is used to minimize the sum of squares of distances from points within the cluster to the center of the cluster. The objective function is as follows:

[0137]

[0138] Among them, J represents the sum of squares of distances from points within the cluster to the cluster center, and p i represents the latent space variable of the reduced-dimensional player, r ik Represents the dimension-reduced player latent space variable p i Whether it belongs to cluster k, o k represents the center of cluster k, ||p i -o k || 2 Represents the dimension-reduced player latent space variable x i The square of the distance to the center of cluster k;

[0139] D2. Assign the reduced-dimensional player latent space variables corresponding to each player to different clusters based on the objective function;

[0140] D3. Calculate the silhouette coefficient of the cluster and compare the silhouette coefficient of the cluster with the preset silhouette coefficient threshold. If the silhouette coefficient of the cluster is greater than the preset silhouette coefficient threshold, it indicates that the reduced-dimensional player latent space variables corresponding to each player are clustered. Otherwise, it indicates that the reduced-dimensional player latent space variables corresponding to each player are not clustered. The calculation formula of the silhouette coefficient is as follows:

[0141]

[0142] Among them, S(i) represents the silhouette coefficient of the cluster, a(i) represents the average distance from the reduced-dimensional player latent space variable to other points in its cluster, and b(i) represents the average distance from the reduced-dimensional player latent space variable to the nearest other clusters.

[0143] It should be noted that the objective function is a function used in the optimization algorithm to guide model learning and classification, and is usually used to measure the quality of a solution. The objective function here aims to minimize the sum of squares of distances from points within the cluster to the center of the cluster, that is, to ensure that the points in the same cluster are gathered as close to the center of the cluster as possible through the optimization algorithm; the silhouette coefficient is an important indicator to measure the clustering effect. The value of the silhouette coefficient ranges from -1 to 1. The closer the value is to 1, the better the clustering result is, and the closer the value is to -1, the poorer the clustering effect is.

[0144] In an optional embodiment, a difficulty adjustment agent is constructed based on the dimensionally reduced player latent space variables, the player's category label, and the historical difficulty sequence, including:

[0145] E1. Construct the state of the difficulty adjustment agent based on the dimension-reduced player latent space variables, the player's category label and the historical difficulty sequence, where the state expression is as follows:

[0146] s t =(z t ,c p ,{d t-τ ,...,d t-1});

[0147] Among them, s t represents the state of the difficulty adjustment agent, c p Indicates the player's category label, {d t-τ ,...,d t-1} represents the historical difficulty sequence;

[0148] E2. Construct the action of the difficulty adjustment agent according to the preset difficulty parameters, where the action expression is as follows:

[0149] a t =[δ enemy ,δ resource , δ puzzle ];

[0150] Among them, a t represents the action of the difficulty adjustment agent, δ enemy Indicates the strength of controlling the enemy AI, δ resource Indicates the richness of control resources, δ puzzle It controls the difficulty of solving puzzles in the game;

[0151] E3. Construct the reward of the difficulty-adjusted agent based on the number of successes and failures of the player's target game, where the reward is expressed as follows:

[0152] r t =0.7r immersion +0.3r frustration ;

[0153] Among them, r t represents the reward of the difficulty-adjusted agent, r immersion represents the immersion level, and r immersion =tanh(2C f -0.5), C f represents the challenge balance coefficient, and tanh represents the hyperbolic tangent function, N success Represents the number of successes of the player in the time window, N attempts Indicates the total number of attempts, N fail represents the number of failures of the player in the time window, r frustration represents the degree of frustration, and r frustration = -log(N fail ).

[0154] In an optional embodiment, the difficulty adjustment agent is adjusted according to the deep deterministic policy gradient to obtain a difficulty adjustment strategy for the target game, including:

[0155] F1, uniformly randomly select actions under a preset probability;

[0156] F2, execute the action and obtain the next state and reward;

[0157] F3, put the current strategy into the experience playback buffer, randomly and evenly collect multiple strategies, and update the main network parameters for each strategy according to the loss function. The trajectory includes the current state, current action, next moment state and reward;

[0158] F4. Update the target network parameters using the main network parameters every preset number of training steps.

[0159] It should be noted that the agent will select an action based on the current state at each moment (each step). The purpose of uniformly random selection of actions is to explore the environment and avoid falling into the local optimal solution. The "uniform random" here means that the probability of each action being selected is equal; the agent selects an action according to the current strategy (which may be random or the optimal action based on the current strategy), and then passes the action to the environment; after executing the action, the environment gives it a new state and a reward based on the behavior of the agent. The reward reflects the feedback given by the environment after the agent takes a specific action in the current state, and is usually used to measure the quality of the agent's behavior; after each interaction with the environment, the agent will store the current state, action, reward and next state (i.e. trajectory) in the experience replay buffer, so that the agent can use historical experience to play Learning can be done without having to rely on the most recent steps every time; as training progresses, a large amount of experience is accumulated in the buffer. By randomly extracting samples from it (batch sampling), the agent can use these historical experiences for training, reducing the impact of time correlation during training; multiple strategies are randomly and uniformly sampled from the experience replay buffer, and these strategies are used to calculate the loss function to update the network parameters. This process is carried out through a deep neural network, and the goal is to optimize the network's behavioral strategy so that the agent can maximize future cumulative rewards; the target network is used to calculate the target Q value to avoid unstable learning process due to fluctuations in the estimated Q value during training. In order to avoid this instability, a target network (target Q network) is usually used to calculate the Q value. The parameters of this network are the same as those of the main network, but the update frequency is lower.

[0160] Embodiment three, as Figure 2 As shown, a game adaptive difficulty adjustment system proposed by the present invention is applicable to the game adaptive difficulty adjustment method, comprising:

[0161] Ability characterization unit 1, which is used to obtain the operation sequence and game status data of the player in the target game, and perform feature fusion on the operation sequence and game status data of the player to obtain the comprehensive ability characteristics of the player;

[0162] The spatial modeling unit 2 is used to perform latent space modeling on the comprehensive ability characteristics of the player according to the variational autoencoder to obtain the player's latent space variables, and perform t-SNE dimension reduction on the player's latent space variables to obtain the reduced dimension player latent space variables;

[0163] The player classification unit 3 is used to classify the reduced-dimensional player latent space variables according to the K-means algorithm to obtain the player's category label;

[0164] An intelligent agent construction unit 4, the intelligent agent construction unit 4 is used to construct a difficulty adjustment intelligent agent according to the dimension reduction player latent space variables, the player's category label and the historical difficulty sequence;

[0165] The difficulty adjustment unit 5 is used to adjust the difficulty adjustment agent according to the deep deterministic policy gradient to obtain the difficulty adjustment strategy of the target game.

[0166] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A method for adaptively adjusting game difficulty, characterized in that: include: Acquire the operation sequence and game status data of the player in the target game, and perform feature fusion on the operation sequence and game status data of the player to obtain the comprehensive ability characteristics of the player; Performing latent space modeling on the comprehensive ability characteristics of the player according to the variational autoencoder to obtain the player latent space variables, and performing t-SNE dimension reduction on the player latent space variables to obtain the reduced dimension player latent space variables; Performing category clustering on the reduced-dimensional player latent space variables according to the K-means algorithm to obtain category labels of the players; Constructing a difficulty adjustment agent according to the reduced-dimensional player latent space variables, the player's category labels, and the historical difficulty sequence; The difficulty adjustment agent is adjusted according to the deep deterministic policy gradient to obtain a difficulty adjustment strategy for the target game.

2. A method for adjusting game difficulty adaptively according to claim 1, characterized in that: in, The operation sequence includes keystroke frequency, reaction time, movement path complexity and combat decision entropy, and the game status data includes level passing time, resource consumption rate and number of failures.

3. A method for adjusting game difficulty adaptively according to claim 2, characterized in that: Get the player's operation sequence and game status data in the target game, including: The key input frequency of the player in the time window is recorded by calculating the number of keys in the preset time window, that is, obtaining the key frequency; Record the time difference from when the player starts to observe the target game state change to when the player performs an operation, that is, obtain the reaction time; Record all positions of the player in the target game to obtain a coordinate point sequence, and calculate the change of position at each moment according to the coordinate point sequence to obtain a differential time series, wherein the expression of the differential time series is as follows: X(t) = U(t) - U(t-1); Among them, X(t) represents the differential time series at time t, and U(t) represents the coordinate point sequence at time t; The cumulative deviation is calculated according to the differential time series, and the fluctuation range and standard deviation are calculated according to the cumulative deviation, wherein the calculation formula of the cumulative deviation is as follows: Where Y(t) represents the cumulative deviation at time t; The calculation formula of the fluctuation range is as follows: R(t)=max(Y(t))-min(Y(t)); Among them, R(t) represents the fluctuation range at time t; The moving path complexity is calculated according to the fluctuation range and the standard deviation, wherein the calculation formula of the moving path complexity is as follows: Among them, C p represents the complexity of the moving path, and S(t) represents the standard deviation at time t.

4. A method for adjusting game difficulty adaptively according to claim 3, characterized in that: Get the player's operation sequence and game status data in the target game, including: Record each decision action of the player and calculate the selection probability of the decision action, wherein the calculation formula of the selection probability is as follows: Among them, P(A i ) represents decision action A i The probability of selection, n i Represents decision action A i The number of times selected, T represents the total number of action decisions; The combat decision entropy of the player is calculated according to the selection probability of the decision action, wherein the calculation formula of the combat decision entropy is as follows: Among them, H d represents the player's combat decision entropy, and N represents the total number of decision actions; Recording the time taken by the player to pass each level in the target game, that is, obtaining the level passing time; Recording the resource consumption rate of the player during the target game, that is, obtaining the resource consumption rate; The number of times the player fails in the target game is recorded, that is, the number of failures is obtained.

5. A method for adjusting game difficulty adaptively according to claim 4, characterized in that: The operation sequence and game status data of the player are subjected to feature fusion to obtain the comprehensive ability features of the player, including: Merging the player's operation sequence and game state data to obtain a merged feature vector, and normalizing the merged feature vector to obtain a normalized merged feature vector; The information entropy of each feature in the normalized merged feature vector is calculated, and the weight of each feature in the normalized merged feature vector is calculated according to the information entropy of the feature, wherein the calculation formula of the weight of the feature is as follows: Among them, ω i represents the weight of the i-th feature in the normalized merged feature vector, α represents the hyperparameter that controls the steepness of the weight distribution, and I(x i ) represents the information entropy of the i-th feature in the normalized merged feature vector; The comprehensive ability feature of the player is determined according to the weight of each feature in the normalized combined feature vector and the normalized combined feature vector.

6. A method for adaptively adjusting game difficulty according to claim 5, characterized in that: The comprehensive ability characteristics of the player are modeled in latent space according to the variational autoencoder to obtain the player latent space variables, including: The mean and standard deviation of the potential distribution of the comprehensive ability characteristics of the player are learned according to the variational autoencoder, wherein the calculation formula of the mean is as follows: μ z NWμ·ReLU(W e F p +b e )4 Among them, μ z represents the mean of the potential distribution of the player’s comprehensive ability characteristics, W μ represents the trainable weight matrix used to generate the mean, W e represents a trainable weight matrix, ReLU represents the activation function, and F p Indicates the comprehensive ability characteristics of the player, b e represents the bias term; The calculation formula of the mean is as follows: Among them, σ z represents the standard deviation of the potential distribution of the player’s comprehensive ability characteristics, W σ represents the trainable weight matrix used to generate the standard deviation; The mean and standard deviation of the potential distribution of the comprehensive ability characteristics of the player are sampled according to the reparameterization to obtain the player latent space variable, wherein the calculation formula of the player latent space variable is as follows: z=μ z +s z ☉∈,∈~N(0,I); Among them, z represents the player latent space variable, ∈ represents the noise sampled from the standard normal distribution, and ⊙ represents element-wise multiplication.

7. A method for adjusting game difficulty adaptively according to claim 6, characterized in that: The reduced-dimensional player latent space variables are clustered according to the K-means algorithm to obtain the category labels of the players, including: Construct an objective function, which is used to minimize the sum of squares of distances from points within the cluster to the center of the cluster, where the objective function is as follows: Among them, J represents the sum of squares of distances from points within the cluster to the cluster center, and p i represents the latent space variable of the reduced-dimensional player, r ik Represents the dimension-reduced player latent space variable p i Whether it belongs to cluster k, o k represents the center of cluster k, ||p i -o k || 2 Represents the dimension-reduced player latent space variable x i The square of the distance to the center of cluster k; Allocating the reduced-dimensional player latent space variables corresponding to each of the players to different clusters based on the objective function; Calculate the silhouette coefficient of the cluster, and compare the silhouette coefficient of the cluster with a preset silhouette coefficient threshold. If the silhouette coefficient of the cluster is greater than the preset silhouette coefficient threshold, it indicates that the reduced-dimensional player latent space variables corresponding to each of the players are clustered. Otherwise, it indicates that the reduced-dimensional player latent space variables corresponding to each of the players are not clustered. The calculation formula of the silhouette coefficient is as follows: Among them, S(i) represents the silhouette coefficient of the cluster, a(i) represents the average distance from the reduced-dimensional player latent space variable to other points in its cluster, and b(i) represents the average distance from the reduced-dimensional player latent space variable to the nearest other clusters.

8. A method for adjusting game difficulty adaptively according to claim 7, characterized in that: Constructing a difficulty adjustment agent according to the dimension-reduced player latent space variables, the player's category label and the historical difficulty sequence, including: The state of the difficulty adjustment agent is constructed according to the dimension-reduced player latent space variables, the player's category label and the historical difficulty sequence, wherein the expression of the state is as follows: s t =(z t ,c p ,{d t-τ ,...,d t-1 }); Among them, s t represents the state of the difficulty adjustment agent, c p Indicates the player's category label, {d t-τ ,...,d t-1 } represents the historical difficulty sequence; The action of the difficulty adjustment agent is constructed according to the preset difficulty parameter, wherein the expression of the action is as follows: a t =[δ enemy ,d resource ,d puzzle ]; Among them, a t represents the action of the difficulty adjustment agent, δ enemy Indicates the strength of controlling the enemy AI, δ tesource Indicates the richness of control resources, δ puzzle It controls the difficulty of solving puzzles in the game; The reward of the difficulty adjustment agent is constructed according to the number of successes and failures of the target game of the player, wherein the expression of the reward is as follows: r t ＝0.7r immersion +0.3r frustration ； Among them, r t represents the difficulty adjustment agent's reward, r immersion represents the immersion level, and r immersion =tanh(2C f -0.5), C f represents the challenge balance coefficient, and tanh represents the hyperbolic tangent function, N success Represents the number of successes of the player in the time window, N attempts Indicates the total number of attempts, N fial represents the number of failures of the player in the time window, r frustration represents the degree of frustration, and r frustration = -log(N fail ).

9. A method for adaptively adjusting game difficulty according to claim 8, characterized in that: The difficulty adjustment agent is adjusted according to the deep deterministic policy gradient to obtain a difficulty adjustment strategy for the target game, including: Uniformly random selection of actions with a preset probability; Execute the action and obtain the next state and reward at the same time; Put the current strategy into the experience playback buffer, randomly and evenly collect multiple strategies, and update the main network parameters for each strategy according to the loss function, where the trajectory includes the current state, current action, next moment state and reward; The target network parameters are updated using the main network parameters every preset number of training steps.

10. A game adaptive difficulty adjustment system, which is applicable to a game adaptive difficulty adjustment method according to any one of claims 9, characterized in that: include: A capability characterization unit (1), the capability characterization unit (1) is used to obtain an operation sequence and game status data of a player in a target game, and to perform feature fusion on the operation sequence and game status data of the player to obtain a comprehensive capability feature of the player; A spatial modeling unit (2), the spatial modeling unit (2) is used to perform latent space modeling on the comprehensive ability characteristics of the player according to the variational autoencoder to obtain the player latent space variables, and perform t-SNE dimension reduction on the player latent space variables to obtain the reduced dimension player latent space variables; A player classification unit (3), the player classification unit (3) is used to classify the reduced-dimensional player latent space variables according to a K-means algorithm to obtain a class label of the player; An intelligent agent construction unit (4), the intelligent agent construction unit (4) is used to construct a difficulty adjustment intelligent agent according to the dimension reduction player latent space variables, the player's category label and the historical difficulty sequence; A difficulty adjustment unit (5), the difficulty adjustment unit (5) is used to adjust the difficulty adjustment agent according to the deep deterministic policy gradient to obtain the difficulty adjustment strategy of the target game.

Images