Game strategy model generation method and behavior strategy generation method of game agent

Abstract

The invention discloses a game strategy model generation method, a game agent behavior strategy generation method, corresponding devices, electronic equipment and a storage medium. The game strategy model generation method comprises the steps of obtaining a preset data set, wherein the preset data set comprises a first state feature sequence of a game agent in a game and a second state feature sequence of a player character; according to the first state feature sequence and the second state feature sequence, training a to-be-trained reward model until a preset training ending condition is met, and obtaining a trained reward model; inputting the target state feature of the game agent into the trained reward model to obtain first reward data corresponding to the game agent; and training a to-be-trained strategy model according to the first reward data, wherein the strategy model is used for generating a behavior strategy for a game agent in the game. According to the method, the stability and scene adaptability of the generated game strategy model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and specifically to a method, device, electronic device, and computer-readable storage medium for generating a game strategy model, and a method, device, electronic device, and computer-readable storage medium for generating a behavior strategy of a game agent. Background Art

[0002] Game characters controlled by electronic devices, or game agents, are widely used in games. Highly skilled and anthropomorphic game agents can significantly enhance the playability and entertainment value of games. In some game scenarios, game agents can engage in combat with player-controlled game characters or perform other actions.

[0003] In related technologies, when using reinforcement learning technology to train game agents, the game agents are controlled by a behavioral strategy model to perform corresponding actions to interact with the game environment, and the reward value of the game agents when performing the action is calculated based on a preset reward function, thereby optimizing the behavior decision network according to the reward value.

[0004] However, in the model training process for generating the agent's strategic behavior, the above-mentioned related technologies iteratively train the strategy model and the reward function simultaneously. That is, the strategy model and the reward function are updated synchronously, which has poor stability and makes it difficult for the model training to converge to the optimal solution.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] In view of this, the present application provides a game strategy model generation method and a game agent behavior strategy generation method, through which the adaptability, robustness and stability of the generated game agent behavior strategy can be improved.

[0007] In a first aspect, an embodiment of the present application provides a method for generating a game strategy model, the method comprising:

[0008] Acquire a preset data set, wherein the preset data set includes a first state feature sequence of a game agent and a second state feature sequence of a player character in the game;

[0009] Training the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is met, thereby obtaining a trained reward model;

[0010] Inputting the target state feature of the game agent in the game into the trained reward model to obtain first reward data corresponding to the game agent;

[0011] The policy model to be trained is trained according to the first reward data.

[0012] In a second aspect, an embodiment of the present application provides a method for generating a behavior strategy of a game agent, the method comprising:

[0013] Obtaining game status data of the game agent in the game;

[0014] The game state data of the game agent is input into the trained strategy model obtained by the game strategy model generation method described in the first aspect, and a behavior strategy generated by the trained strategy model according to the game state data of the game agent is obtained.

[0015] In a third aspect, an embodiment of the present application provides a device for generating a game strategy model, the device comprising:

[0016] Acquisition unit, training unit and processing unit;

[0017] The acquisition unit is used to acquire a preset data set, wherein the preset data set includes a first state feature sequence of a game agent and a second state feature sequence of a player character in the game;

[0018] The training unit is configured to train the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is reached, thereby obtaining a trained reward model;

[0019] The processing unit is configured to input the target state feature of the game agent in the game into the trained reward model to obtain first reward data corresponding to the game agent;

[0020] The training unit is further configured to train the strategy model to be trained according to the first reward data.

[0021] In a fourth aspect, an embodiment of the present application provides a device for generating a behavior strategy of a game agent, the device comprising: an acquisition unit and a generation unit;

[0022] An acquisition unit, configured to acquire game state data of the game agent in the game;

[0023] A generation unit is used to input the game status data of the game agent into the trained strategy model obtained by the game strategy model generation method described in the first aspect, and obtain the behavioral strategy generated by the trained strategy model according to the game status data of the game agent.

[0024] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0025] processor; and

[0026] The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method of the first aspect or the second aspect is executed.

[0027] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing a data processing program, which is run by a processor to execute the method of the first aspect or the second aspect.

[0028] The game strategy model generation method and the game agent strategy generation method provided in the present application can obtain a preset data set, which includes a first state feature sequence of the game agent in the game and a second state feature sequence of the player character; according to the first state feature sequence and the second state feature sequence, the reward model to be trained is trained until the preset training end condition is reached to obtain a trained reward model; based on the trained reward model, the target state characteristics of the game agent in the game can be input into the trained reward model to obtain a relatively stable first reward data of the game environment in real time feedback to the game agent; and then the strategy model to be trained can be trained according to the first reward data, and the strategy model is a model used to generate behavioral strategies for the game agent in the game.

[0029] Since the generation process of the above-mentioned strategy model is to train the strategy model based on the trained reward model after training the reward model, the strategy model can be generated based on the optimal and more instructive reward data obtained from the trained reward model, rather than using less than ideal reward data to train the strategy model in related technologies. In this way, the training process of the strategy model is more stable, and generating the strategy model through more instructive reward data helps the strategy model to understand and learn the desired behavior strategy more quickly when generating the strategy model, thereby achieving a better anthropomorphic effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 A flowchart of a method for generating a game strategy model according to an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a process for determining target team status characteristics provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a flow chart for determining the reward value at each time point provided in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of a flow chart for determining the reward value at each time point based on historical team status characteristics provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of a reward model provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a process for determining second reward data provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of an example of the network structure of the strategy model provided in an embodiment of the present application.

[0038] Figure 8 This is a flowchart of an example of fine-tuning the reward model and strategy model when updating the game version provided in an embodiment of the present application;

[0039] Figure 9 A schematic diagram of a training structure of the LoRA model provided in an embodiment of the present application;

[0040] Figure 10 A flowchart of a method for generating a behavioral strategy for a game agent provided in an embodiment of the present application;

[0041] Figure 11 A schematic diagram of the structure of a game strategy model generation device provided in an embodiment of the present application;

[0042] Figure 12 A schematic diagram of the structure of a device for generating a behavior strategy for a game agent provided in an embodiment of the present application;

[0043] Figure 13 An example electronic device structural block diagram is provided for generating a game strategy model or generating a behavior strategy of a game agent in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0045] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including", "having" and their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "Including A, B and / or C" means including any one, any two, or any three of A, B, and C.

[0047] It should be understood that in the embodiments of the present application, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0048] Before introducing the game strategy model generation method provided by the present application in detail, the relevant concepts involved in the embodiments of the present application are first introduced.

[0049] 1. Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. By simulating, extending, and expanding human intelligent behavior, AI enables analysis, decision-making, and resolution of complex problems. AI encompasses multiple fields, including machine learning and deep learning.

[0050] 2. An agent refers to an entity in the field of artificial intelligence that can perceive the environment, make decisions, and perform actions. The entity can be a physical entity (such as a robot) or a virtual entity (such as a virtual character in a game or a software program).

[0051] 3. Game Agent refers to an intelligent entity (such as a non-player character) that runs in a game environment and has the ability to make autonomous decisions and take actions. The game agent perceives the state and information of the game environment, such as maps, enemy positions, props, etc., and makes decisions based on the set goals, rules and strategies, choosing the best actions to achieve the game's victory conditions or complete specific tasks. In the game, the game agent can play different game roles, such as game characters of the enemy camp, teammate characters, non-player characters, etc. The behavior and degree of anthropomorphism of the game agent in the game scene directly affect the difficulty, realism and fun of the game. In this embodiment, a relevant model of the game agent's behavior strategy can be generated based on AI design.

[0052] 4. Reinforcement Learning (RL) is a self-supervised learning method for solving sequential decision-making problems. It enables game agents to learn autonomously without the need for manual training data collection. In RL, the game agent observes the current state of the game environment, selects an action to influence the environment based on this state, and receives a reward signal to evaluate the quality of this action. The goal of the game agent is to gradually learn the optimal strategy through interaction with the game environment, by trying different actions, and based on the reward signal. In other words, it is able to choose the best action given the game environment state. The core concept of RL is based on the optimization of a reward function or a policy function. The reward function represents the expected reward of the game agent choosing a certain action under given game environment state, while the policy function represents the probability of the game agent choosing an action under given environment state. By continuously updating and optimizing these functions, the game agent can gradually improve the quality of its decision-making.

[0053] For example, when training a game-playing agent, the reward function can be designed to give a high reward when the agent successfully completes a level, a moderate reward when it makes progress in the game, and a negative reward when the agent's health decreases or the game fails. In this way, the agent will learn how to obtain as many rewards as possible through repeated attempts, thereby learning how to play the game better.

[0054] 5. Imitation Learning (IL) is a non-reinforcement learning machine learning method that generates policies by learning from the behavior of humans or other intelligent agents. The main idea is to solve specific tasks or problems by observing and learning from the behavior of others. In imitation learning, training data typically comes from behavioral records of humans or other intelligent agents. This data is used to train the model to generate behaviors similar to the original ones. Unlike reinforcement learning, imitation learning does not require interaction with the environment or setting a reward function. Therefore, imitation learning makes it easier to achieve the target task, generates policies that are more consistent with human behavior, and avoids some problems that may arise in reinforcement learning, such as training instability and low data efficiency.

[0055] 6. Generative Adversarial Imitation Learning (GAIL) is a novel imitation learning method for learning the behavior of game-playing agents from expert examples. It combines generative adversarial networks (GANs) and reinforcement learning techniques. GAIL uses a GAN to train a generator to produce state-action pairs similar to expert trajectories, and then uses policy iteration in reinforcement learning for training. This enables game-playing agents to learn better policies from expert examples without explicitly defining a reward function.

[0056] With the rapid development of online games, an increasing number of game modes require players to possess high-level strategic thinking, teamwork, and real-time decision-making abilities. Consequently, it is necessary to train and optimize online gaming agents to develop these skills. However, traditional methods for training gaming agents often fall short in terms of training stability, strategic thinking, teamwork, and real-time decision-making abilities.

[0057] Below, the prior art involved in this application and the problems existing in the prior art are described:

[0058] Currently, traditional training methods for game agents may include but are not limited to the above-mentioned reinforcement learning method, imitation learning method, and generative adversarial model learning method.

[0059] As mentioned above, in reinforcement learning, a game agent observes the current state of the game environment, selects an action based on that state to influence the environment, and receives a reward signal to evaluate the effectiveness of that action. The training goal of the game agent is to gradually learn the optimal strategy through interaction with the game environment, by trying different actions, and based on the reward signals received for each action. Specifically, the game agent is able to select the optimal action given the state of the game environment.

[0060] During the model training process for generating the agent's strategic behavior, this related technology iteratively trains the policy model and reward function simultaneously. That is, the policy model and reward function are updated synchronously, resulting in poor stability and difficulty in model training converging to the optimal solution.

[0061] While carefully adjusting the reward function of reinforcement learning algorithms can address the problem of some agents' behavior not being sufficiently similar to human behavior, the inherent subjectivity and complexity of anthropomorphism—for example, human behavior and decision-making are often influenced by multiple factors (such as emotions, values, and cultural background)—makes it difficult to quantify these factors when designing reward functions for reinforcement learning. Furthermore, human behavior often requires real-time adjustments based on specific scenarios, making it difficult to fully cover all game scenarios. Consequently, reward function design is complex, making it difficult to effectively optimize the behavior of game agents.

[0062] As mentioned above, it is difficult to design a comprehensive reward function through reinforcement learning, which makes it difficult to optimize the behavior of game agents.

[0063] Based on this, in related technologies, it is also possible to train game agents by adopting imitation learning methods that are not based on reward functions. Imitation learning methods that are not based on reward functions can be referred to in the previous article and will not be repeated here.

[0064] However, in collaborative team games, the data that game agents need to imitate is often the sequential behaviors of multiple agents interacting with each other, which are often highly dimensional and complex. These complex interactions can dramatically increase the state space, action space, and possible behavior patterns, significantly increasing the dataset size required for imitation learning and making it difficult to fully capture all possible scenarios from limited expert examples. Furthermore, game agents must not only understand the behavior patterns of individual agents but also master collaborative strategies across multiple agents. This collaborative behavior often relies on communication, role division, and goal coordination among team members, which is difficult to extract directly from a single-viewpoint action sequence during imitation learning.

[0065] Therefore, gaming agents trained using imitation learning methods have poor teamwork, understanding of team member intentions, and competitive balance in competitive games. Failure to promptly understand teammates' intentions can easily lead to team members learning to execute only one game action, resulting in an inability to coordinate combat strategies. As can be seen, the performance of gaming agents trained using imitation learning methods is often limited by the complexity of teamwork. As the number of team members increases, the model's error can grow exponentially, resulting in poor teamwork imitation.

[0066] Alternatively, game agents can be trained using generative adversarial imitation learning. As previously mentioned, this method theoretically offers some advantages over imitation learning. However, in practice, it suffers from instability in the training process and difficulty aligning the capabilities of the discriminator and generator. If the discriminator is too powerful, learning strategies that mimic human behavior becomes difficult, leading to unnatural behavior or over-strengthening of certain state characteristics when mimicking human behavior, thus affecting the player's gaming experience. Furthermore, generative adversarial imitation learning typically relies on a large amount of high-quality expert data. Insufficient or unevenly distributed data can also affect training stability and final performance.

[0067] The above exemplary methods often only require retraining the game agent when the game version changes. However, due to the complexity of the game scene, training the game agent requires a lot of time and cost, and retraining the game agent will significantly affect the launch time of the game.

[0068] To address at least some of the above-mentioned problems, the present application provides a method for generating a game strategy model, a corresponding device for generating a game strategy model, an electronic device capable of implementing the method, and a computer-readable storage medium, as well as a method for generating a behavior strategy for a game agent, a corresponding device for generating a behavior strategy for a game agent, an electronic device capable of implementing the method, and a computer-readable storage medium. The following examples provide detailed descriptions of the above-mentioned methods, devices, electronic devices, and computer-readable storage media.

[0069] The game strategy model generation method and the game agent behavior strategy generation method provided in the embodiments of the present application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, etc. The terminal can also include a client, which can be a game application client, a browser client with a game program, or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0070] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0071] like Figure 1 As shown, Figure 1 This is a flowchart of an example of a method for generating a game strategy model provided in an embodiment of the present application. It should be noted that the steps shown can be performed in a different logical order than that shown in the flowchart of the method. The method may include the following steps S110 to S140.

[0072] Step S110: Obtain a preset data set, which includes a first state feature sequence of a game agent and a second state feature sequence of a player character in the game.

[0073] The above-mentioned game agent can be any one or more game agents controlled by electronic devices in the game, such as non-player characters (NPCs), and there is no specific limitation on this.

[0074] The player character (PC) is a virtual character controlled by a human player. This player character can be pre-screened to meet preset conditions for imitation by the game agent. These pre-screened conditions may include, but are not limited to, high-level operation, high-level game play, and a more interesting strategy style in the game. By training the reward model with the state characteristics of the screened player characters, the reward model can learn higher-quality anthropomorphic behaviors.

[0075] The above-mentioned state feature sequence can be understood as a sequence composed of the game state features of the game agent or player character at multiple time nodes, that is, a plurality of state features can be included in the state feature sequence. The so-called first state feature sequence and second state feature sequence are only used to distinguish the state feature sequences of different objects, and are not used to describe a specific order or sequence. In addition, this embodiment does not limit the number of the above-mentioned first state feature sequences and second state feature sequences obtained, and multiple first state feature sequences and multiple second state feature sequences can be obtained.

[0076] The above-mentioned state features are a set of key attributes or indicators used to describe the game state of the game agent or player character at a certain time point. The state features may include but are not limited to the position, speed, blood volume, life value, energy value, number of surrounding teammates or enemies, game terrain, game gain, etc. of the game agent or player character. It should be noted that in this embodiment, the state feature sequence is used to train the reward model. The adjacent state features in the state feature sequence can characterize what game actions the game agent or player character has performed. For example, the two adjacent state features of the game agent respectively record the two corresponding positions of the game agent in the game scene. It can be determined by the two positions that the game agent has performed a moving action. Therefore, in this embodiment, the above-mentioned state features do not include the action data performed by the game agent or player character.

[0077] The above-mentioned preset dataset is composed of the state feature sequences of the game agent and the player character during the historical game process collected in advance. In other words, the training of the reward model is offline training using the training dataset.

[0078] It can be understood that the first state feature sequence of the game agent and the second state feature sequence of the player character can be the state feature sequences of the game agent and the player character in the same game scene in the game. The first state feature sequence and the second state feature sequence can be regarded as a sample pair in the training data set prepared in advance for training the reward model to be trained. The reward model to be trained is trained with the game agent as the negative sample in the sample pair and the player character as the positive sample in the sample pair. Among them, the second state feature sequence of the player character is used as a positive sample to represent the state feature that can obtain a good reward, and the first state feature of the game agent is used as a negative sample to represent the state feature that obtains a poor reward. When training the reward model later, the difference between the reward data obtained by the positive and negative samples can be used to adjust the parameters of the reward model so that the reward model can maximize the imitation of the behavioral state of the player character.

[0079] It should be noted that in games, the state characteristics corresponding to a game agent can include its own state characteristics in the game environment as well as the state characteristics of other game agents or player characters associated with the game agent. This association can be understood as the game agent related to the action performed by the game agent under a certain state characteristic, or the virtual character that belongs to the same team as the game agent, that is, the game agent is a teammate. For example, if game agent A performs the game behavior action of shooting game agent B, the state characteristics corresponding to game agent A may include attribute information such as the location and health value of game agent B.

[0080] Step S120: The reward model to be trained is trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is reached, thereby obtaining a trained reward model.

[0081] The aforementioned preset training termination conditions may include, but are not limited to: the loss value of the loss function of the reward model to be trained is less than or equal to a preset threshold, the number of training times reaches a preset number, the reward value output by the reward model is greater than or equal to a preset reward value, etc. This embodiment does not specifically limit this.

[0082] Step S130: Input the target state characteristics of the game agent in the game into the trained reward model to obtain the first reward data corresponding to the game agent.

[0083] In this embodiment, the strategy model is trained online, allowing the game agent to adjust its strategy generation in real time based on the player's behavior. Therefore, the target state features described above can be understood as the current state features of the game agent acquired in real time during strategy model training. These target state features may include the state features in the first state feature sequence described above.

[0084] This step determines the first reward data corresponding to the target state characteristics of the game agent based on the trained reward model. This step applies the trained reward model. Note that the first reward data is the optimal reward data given by the trained reward model for the target state characteristics of the game agent.

[0085] Step S140: training the policy model to be trained according to the first reward data.

[0086] The behavioral strategy generated for a game agent refers to the decision-making rules that the game agent uses to decide what actions to take based on the state information it perceives in the game environment.

[0087] In this embodiment, the network structure of the policy model is not specifically limited. For specific examples, please refer to the detailed description below.

[0088] This embodiment optimizes the strategy model so that the trained strategy model can generate anthropomorphic behavioral strategies for the game agent and control the game agent to perform human-like actions through the anthropomorphic behavioral strategies. Based on the trained reward model, the reward data provided by the game environment to the game agent during interaction with the game environment tends to be stable. Therefore, when the strategy model is trained based on the first reward data, the stability of the training process can be improved.

[0089] In addition, a strategy model is generated based on the optimal and more instructive reward data obtained from the trained reward model, rather than the less than ideal reward data training strategy model used in related technologies. In this way, the strategy model can be generated through more instructive reward data to help the game agent strategy model understand and learn the desired behavior strategy more quickly when generating the strategy model, thereby achieving a better anthropomorphic effect.

[0090] In some optional embodiments, in the above step S120, "training the reward model to be trained according to the first state feature sequence and the second state feature sequence until the preset training end condition is reached to obtain the trained reward model" can be specifically implemented through the following steps S121 and S122.

[0091] Step S121: Input the first state feature sequence and the second state feature sequence into the reward model to be trained to obtain the second reward data corresponding to the game agent and the third reward data corresponding to the player character.

[0092] The reward model is a reward function that defines the feedback an agent or player character receives from the game environment after performing an action. Specifically, the reward model quantifies the game environment's immediate evaluation of the agent's actions and expresses it numerically as a "reward."

[0093] It should be noted that the above-mentioned second reward data is the reward data obtained during the training of the reward model, and the first reward data introduced above is the optimal reward data output by the reward model after training, that is, the optimal reward data obtained by inputting the first state feature sequence when the reward model parameters are optimal.

[0094] Step S122: Adjust the parameters of the reward model to be trained according to the second reward data and the third reward data until a preset training end condition is reached to obtain a trained reward model.

[0095] As previously described, the first-state feature sequence and the second-state feature sequence can constitute positive and negative sample pairs. Specifically, the training process for the reward model to be trained in this embodiment is as follows: input the positive and negative sample pairs into the reward model to be trained, obtain reward data, and then adjust parameters based on the reward data. It is understood that this training process is an iterative training process. That is, if the reward model after parameter adjustment does not meet the preset training end condition, the training process returns to the reward model after the parameter adjustment, and the training continues until the preset training end condition is met, completing the training of the reward model.

[0096] In a specific embodiment, the process of inputting the first state feature sequence into the reward model to be trained in step S121 to obtain the second reward data corresponding to the game agent specifically includes the following steps S1210 and S1211. It will be understood that in this embodiment, the process of inputting the first state feature sequence into the reward model to be trained to obtain the second reward data is consistent with the process of inputting the second state feature sequence into the reward model to be trained to obtain the third reward data. Therefore, in the subsequent description of the solution, the second reward data corresponding to the first state feature sequence will be described.

[0097] Step S1210: Determine the reward value corresponding to the state feature of each time node in the first state feature sequence of the game agent through the reward model to be trained.

[0098] Step S1211: The sum of the reward values ​​corresponding to each time node is determined as the second reward data fed back by the game environment to the game agent.

[0099] The above-mentioned time nodes can be understood as moments with specific time intervals. In the embodiment of the present application, a time node is a specific moment when the game agent interacts with the game environment. A time node may be quantified as a specific date, a timestamp, or simply an abstract time step index (such as the time unit moved after each decision step). For example, if the game agent performs an action every 150 milliseconds in the game environment, the time interval between each time node is 150 milliseconds.

[0100] It is understandable that the time intervals between the above-mentioned time nodes can be customized, and this embodiment does not limit this.

[0101] After determining the time nodes at which the game agent interacts with the game environment, we can record and analyze the state and reward data corresponding to the game agent at these time nodes, and then use algorithms to guide the game agent to optimize its behavior strategy.

[0102] As previously explained, a state feature sequence is a sequence of state features corresponding to multiple time nodes. Therefore, in this specific embodiment, the reward model can generate a corresponding reward value for each state feature at each time node in the state feature sequence, and then sum the reward values ​​corresponding to multiple time nodes to obtain the final reward data.

[0103] This allows for a training method that trains the reward model based on the state feature sequence corresponding to the game agent, rather than just a single state feature at a specific time point. In other words, this embodiment calculates the reward data corresponding to the entire state feature sequence, determining whether the global state features of the game agent are closer to those of a human player, rather than determining whether a local state feature is closer to that of a human player.

[0104] In this way, considering the degree of anthropomorphism of the state characteristics of the game agent based on the global perspective can improve the generalization ability of the anthropomorphism of the NPC behavior trajectory, find deeper correlations between various behavioral characteristics, enable the reward model to better understand and capture long-term dependencies and dynamic changes in complex environments, and make the NPC's state characteristics more natural, without abrupt changes in the state feature sequence where some state characteristics look like humans and some do not look like humans.

[0105] For example, assuming that a state feature sequence includes state features corresponding to N time nodes, the reward data R corresponding to the state feature sequence can be calculated using the following formula (1).

[0106]

[0107] Among them, R can be used to represent the reward data output by the reward model to be trained; r i Represents the reward value corresponding to the state feature of the i-th time node in the state feature sequence; N is the number of time nodes in the state feature sequence; i represents a time node in the time nodes of the state feature sequence, and i can traverse 1 to N.

[0108] In this embodiment, if the first state feature sequence is input for the reward model to be trained, the second reward data corresponding to the first state feature sequence of the game agent can be expressed as R NPC If the second state feature sequence is input for the reward model to be trained, the third reward data corresponding to the second state feature sequence of the player character can be expressed as R 人类 .

[0109] The following describes the specific details of determining the reward value at each time node by using the reward model to be trained in step S1210.

[0110] In order to address the defect of the prior art that it is only applicable to 1v1 game scenarios, the game strategy model generated in the embodiment of the present application is mainly aimed at multi-player versus multi-player battle scenarios. Therefore, the game in the above step S110 can be a team combat game, which can include at least one camp. In the game, camps refer to different groups or organizations, which may form opposition, alliance or neutral relationships based on the game story background, goals or concepts. In a multi-player versus multi-player battle scenario, a camp can include multiple virtual characters, which can include game agents and / or player characters. Game agents and player characters in the same camp can act as teammates to fight in a team.

[0111] Based on the above game scenario, the above step S1210 can be specifically implemented through the following steps S01 to S03.

[0112] Step S01: Determine the first state feature corresponding to the i-th time node in the first state feature sequence of the game agent.

[0113] The above i traverses 1 to N, where N is an integer and is the number of time nodes included in the first state feature sequence. The above first state feature is the specific state feature corresponding to a time node in the first state feature sequence. For example, when i = 1, the first state feature is the state feature corresponding to the first time node.

[0114] Step S02: According to the first state feature corresponding to the i-th time node, determine the target team state feature corresponding to the i-th time node of the camp to which the game agent belongs.

[0115] The above-mentioned target team status characteristics can be understood as a series of data and indicators that reflect the overall status of the team of the camp to which the game agent belongs at the i-th time node; for example, the position of each virtual character in the camp, the health value of each virtual character, the combat status of each virtual character, etc.

[0116] That is, this step S01 is used to determine the target team state characteristics corresponding to the camp to which it belongs at the i-th time node based on the first state characteristics of the game agent at the i-th time node.

[0117] In a specific embodiment, step S01 can be implemented through the following steps A1 to A2.

[0118] Step A1: Determine the first target feature corresponding to the game agent in the first state feature corresponding to the i-th time node and the second target feature corresponding to each virtual character in the camp except the game agent; wherein the first target feature is used to represent the game state feature corresponding to the game agent at the i-th time node, and the second target feature is used to represent the game state feature corresponding to the virtual character at the i-th time node.

[0119] As mentioned above, in team combat games, the state characteristics of the game agent at each time node are closely related to the teammates in its camp. Therefore, the state characteristics corresponding to the game agent at each time node may include the state characteristics of the game agent itself, as well as the state characteristics of other game agents or player characters associated with the game agent.

[0120] In this embodiment, the target team status characteristics of the camp to which the game agent belongs can be determined by considering the correlation between the status characteristics of the game agent itself and the status characteristics of teammates. Therefore, this step is used to distinguish the first target status characteristics belonging to the game agent itself in the first status characteristics and the second target status characteristics of the teammates of the game agent.

[0121] Step A2: Determine the target team status feature corresponding to the camp to which the game agent belongs at the i-th time node based on the first target feature and the second target feature.

[0122] In a specific embodiment, correlation data between the first target feature and the second target feature can be determined, and then the target team status feature corresponding to the camp to which the game agent belongs at the i-th time point can be determined based on the correlation data. In other words, by determining the dependencies and correlations between the first target feature and each second target feature, the reward model can consider the status information of teammates when predicting the game agent's own reward.

[0123] like Figure 2 As shown, determining the correlation data between the first target state feature and the second target state feature and determining the target team state feature based on the correlation data can be implemented based on a Transformer network unit. Specifically, the first target state feature and each second target state feature at the i-th time node are input into the Transformer network unit, which then outputs the target team state feature through a multi-head attention mechanism.

[0124] Step S03: Encode the target team state features corresponding to the i-th time node to obtain the reward value fed back by the game environment to the game agent at the i-th time node.

[0125] After determining the target team status characteristics through step S02, the reward model can encode the target team status characteristics in a preset manner to obtain the reward value of the game agent at the i-th time node.

[0126] In a specific embodiment, Figure 3As shown, the target team state features can be input into a fully connected neural network (Multi-Layer Perception, MLP), or a multi-layer perceptron for encoding, and the reward value of the game agent at the i-th time node can be output.

[0127] By determining the target team's state characteristics, we can dynamically predict the reward value given to the camp by the game environment based on the changes in the team's state characteristics. In this way, the parameters of the reward model to be trained can be adjusted based on the reward value. The advantages of training the model to be trained based on team state characteristics include the following:

[0128] First, it enables the reward model to be trained to dynamically reward and punish optimal behaviors in different situations. For example, in a multiplayer game, when the team's health is generally low, the reward model can increase the reward for healing behaviors; while when it is necessary to concentrate firepower to attack the enemy leader, the reward model can increase the reward for damage-dealing behaviors.

[0129] Second, adjusting the reward model to be trained based on the reward value calculated based on the target team's state characteristics can enable the adjusted reward model to be trained to strengthen the coordination within the camp. If the reward model can identify behaviors that are beneficial to the overall interests of the team and give higher rewards, then each game agent will be more inclined to adopt a cooperative strategy rather than focusing solely on individual achievements.

[0130] Third, understanding the team's state characteristics can help the reward model better guide resource utilization. For example, when resources are limited, the reward model can adjust the reward distribution based on the current team needs, ensuring that key resources are properly utilized when they are most needed.

[0131] In summary, by adjusting the parameters of the reward model to be trained based on the target team state characteristics, a more refined, flexible, and effective incentive mechanism can be achieved. This has a positive impact on cultivating a good team collaboration atmosphere, optimizing decision-making, and improving the overall performance of the game, thereby creating a more balanced, interesting, and challenging game environment and improving the player's team combat gaming experience.

[0132] In an optional specific embodiment, before encoding the target team status characteristics to obtain the reward value, the target team status characteristics can also be combined with the historical team status characteristics of the camp where the game agent is located. In this way, the reward model to be trained can refer to the historical team status characteristics to accurately understand the status characteristics of the camp where the game agent is located at the i-th time node, such as what actions the camp has taken in the past and what changes it has experienced, etc., which helps to accurately evaluate the reward data corresponding to the status characteristics of the i-th time node.

[0133] At the same time, referring to historical team state characteristics can reduce the changes in state characteristics caused by single observations or accidental events, improve the stability and consistency of the second reward data generated by the reward model to be trained, and make the learning process of the reward model to be trained more stable.

[0134] In addition, since historical team state features can contain different but related game scenarios, the reward model can draw on past experience to adapt to new environments or tasks more quickly, achieving a certain degree of transfer learning, that is, improving the learning speed of the reward model to be trained.

[0135] Specifically, the target team status characteristics may be determined according to the following steps S04 to S05.

[0136] Step S04: Obtain the historical team status characteristics corresponding to the camp to which the game agent belongs at the i-1th time node.

[0137] Step S05: Determine the historical team status characteristics corresponding to the camp to which the game agent belongs at the i-th time node based on the historical team status characteristics and the target team status characteristics.

[0138] That is, after calculating the target team status characteristics at the i-th time node, the historical team status characteristics at the i-1-th time node are obtained. It can be understood that when i=1, it can indicate that the camp where the game agent belongs is newly formed and has no team status characteristic information.

[0139] By mixing the historical team status features at the i-1th time node with the target team status features at the i-th time node, the historical team status features of the camp to which the game agent belongs at the i-th time node can be generated.

[0140] Correspondingly, in the above step S03, "encoding the target team status characteristics corresponding to the i-th time node, and obtaining the reward value fed back by the game environment to the game agent at the i-th time node" can specifically be encoding the historical team status characteristics corresponding to the i-th time node, and obtaining the reward value fed back by the game environment to the game agent at the i-th time node.

[0141] like Figure 4As shown, the historical team state features at the i-1th time node and the target team state features at the i-th time node can be input into a gated recurrent neural network (GRU). The GRU network unit can mix the historical team state features at the i-1th time node and the target team state features at the i-th time node to output the historical team state features corresponding to the i-th time node. The historical team state features corresponding to the i-th time node output by the GRU network unit are then input into the MLP network unit for encoding to obtain the reward value at the i-th time node after reference to the historical team state information.

[0142] Based on the above introduction, it can be seen that in some specific embodiments, such as Figure 5 As shown, Figure 5 A schematic diagram of the network structure of a reward model provided in an embodiment of the present application. The reward model constructed for the team collaboration game scenario in this application may include a Transformer network unit, a GRU network unit, and an MLP network unit. After decomposing the first state feature of the i-th time node to obtain the first target state feature corresponding to the game agent and the second target state feature corresponding to the remaining virtual characters, according to Figure 5 The reward model shown can calculate the reward value corresponding to the game agent at the i-th time node.

[0143] When i traverses 1 to N, such as Figure 6 As shown, at each time node, the reward value corresponding to the time node is calculated by the reward model, and then the reward values ​​of each time node are summed up to obtain the second reward data corresponding to the first feature sequence of the game agent.

[0144] Similarly, the third reward data corresponding to the second feature sequence of the player character can be determined based on the above-mentioned method of solving the second reward data corresponding to the game agent, which will not be repeated here.

[0145] In this embodiment, the main idea of ​​adjusting the parameters of the reward model to be trained according to the second reward data and the third reward data is: adjusting the parameters of the reward model to be trained based on the principle of making the second reward data close to the third reward data, that is, when the difference between the second reward data and the third reward data reaches a preset difference, it indicates that the reward model to be trained has learned how to judge whether the state characteristics of the game agent are close to the player character.

[0146] Based on the above adjustment principles, this embodiment constructs a loss function as shown in formula (2) for the reward model.

[0147] loss=-E[log(σ(R 人类 -R NPC))] (2)

[0148] Among them, loss is used to represent the loss value of the reward model loss function; R 人类 Used to indicate the third reward data; R NPC Used to represent the second reward data; σ represents the sigmoid activation function; E[] represents the expected value. 人类 -R NPC Used to represent the difference between the third reward data corresponding to the player character's behavioral feature sequence and the second reward data corresponding to the NPC's behavioral feature sequence;

[0149] σ(R 人类 -R NPC ) is used to map the gap between the third reward data and the second reward data to the interval (0, 1), which can be used to measure the probability value of the degree of advantage of the player character's status characteristics compared to the NPC's status characteristics; specifically, when the third reward data R 人类 With the second reward data R NPC The larger the difference between them and the positive value is, the greater the σ(R 人类 -R NPC ) is closer to 1, indicating that the player character's status characteristics have a higher score or advantage than the NPC's status characteristics; when the third reward data R 人类 With the second reward data R NPC The difference between them is small and negative, then σ(R 人类 -R NPC ) is closer to 0, indicating that the NPC's status characteristics are comparable to or more advantageous than those of the player character.

[0150] Combined with formula (2), that is, in this embodiment, the adjustment of the parameters of the trained reward model based on the second reward data and the third reward data in step S122 can be specifically implemented as follows: based on the second reward data and the third reward data, a loss value of the loss function corresponding to the reward model to be trained is determined; then, based on the principle that the loss value is less than a first preset threshold, the parameters of the reward model to be trained are adjusted to obtain a trained reward model. That is, when the loss value is less than the first preset threshold, it indicates that the reward model to be trained has completed training.

[0151] It is understandable that if the loss value is required to be smaller, it is necessary to make E[log(σ(R 人类 -R NPC ))] is larger, that is, it is necessary to make σ(R 人类 -R NPC ) The greater the probability value output, the more specific the third reward data R 人类 With the second reward data R NPCIn other words, by adjusting the difference between the third reward data corresponding to the state characteristics of the player character and the second reward data corresponding to the state characteristics of the NPC, the reward model can be trained to imitate the behavior of the player character.

[0152] After the reward model is trained, the strategy model can be further optimized based on the reward model. The strategy model is a model used to generate behavioral strategies for game agents in the game. In layman's terms, it is a model that determines what actions the game agent performs in what state.

[0153] In an optional specific embodiment, the above step S140 can be implemented through the following steps S141 to S142.

[0154] Step S141: Input the target state characteristics of the game agent in the game into the strategy model to be trained to obtain the predicted action probability distribution.

[0155] Step S142: training the policy model to be trained based on the first reward data and the predicted action probability distribution.

[0156] In some specific embodiments, Figure 7 As shown, Figure 7 A network diagram of a strategy model is provided. The strategy model may include a Transformer network unit, a GRU network unit, and an MLP network unit.

[0157] Although the network structure of the strategy model is similar to that of the reward model introduced above, the two are inconsistent in training objectives, output data and uses. Among them, the reward model is mainly used to predict or estimate the reward obtained after taking a certain action under a given state feature, that is, input state features and action features, and output the reward value. The strategy model is used to select the appropriate action under a given state feature, that is, input state features, and output the action probability distribution of multiple executable actions. That is, after obtaining the first reward data corresponding to the target state feature of the game agent based on the trained reward model, the target state feature can be input into the strategy model to be trained, and the strategy model to be trained can output the predicted action probability distribution of each action that the game agent can perform under the target state feature for the target state feature. The above-mentioned predicted action probability distribution is used to guide the game agent to select the action with the highest probability as the target action to be executed next and execute the target action at the corresponding time node.

[0158] In a specific embodiment, the above step S142 can be implemented through the following steps S1420 to S1421.

[0159] Step S1420: Determine the similarity between the predicted action probability distribution and the preset action probability distribution for the game agent.

[0160] Step S1421: Determine the benefit value of the benefit function corresponding to the strategy model to be trained based on the first reward data and the similarity, and adjust the parameters of the strategy model to be trained according to the principle that the benefit value is greater than or equal to the second preset threshold.

[0161] The above-mentioned preset action probability distribution is the probability distribution of game actions that are pre-designed for the game agent and have a low degree of anthropomorphism. In the embodiment of the present application, the preset action probability distribution can be the initial action probability distribution obtained by inputting the target state action when the strategy model to be trained is not trained (the model parameters are not optimized).

[0162] The similarity between the above-mentioned predicted action probability distribution and the preset action probability distribution for the game agent is a quantitative indicator to measure the predicted action probability distribution and the preset action probability distribution, that is, it is used to describe the degree of match between the predicted action probability distribution and the preset action probability distribution.

[0163] The higher the similarity between the predicted action probability distribution and the preset action probability distribution for the game agent, the better the predictive ability of the policy model.

[0164] In a specific embodiment, the similarity between the above-mentioned predicted action probability distribution and the preset action probability distribution can be calculated by KL divergence (Kullback-Leibler Divergence). KL divergence measures the similarity between two distributions. KL divergence, also known as relative entropy, is an asymmetric measurement method commonly used to measure the distance between two probability distributions. KL divergence can also measure the distance between two random distributions. The higher the similarity of two random distributions, the smaller their KL divergence. When the difference between the two random distributions increases, their KL divergence will also increase. The KL divergence can be used to limit the degree of optimization of the strategy model to avoid over-optimization during the training of the strategy model.

[0165] The benefit value of the benefit function corresponding to the aforementioned strategy model can be understood as quantifying the benefit gained by the player character or game agent after adopting a certain strategy. In this embodiment, when adjusting the parameters of the strategy model, the adjustment can be performed so that the benefit value is greater than or equal to a second preset threshold. That is, when the benefit value is equal to or greater than the second preset threshold, it indicates that the strategy model has reached an optimal level.

[0166] In this embodiment, the Proximal Policy Optimization (PPO) algorithm can be used to adjust the policy model's parameters. Its core idea is to maximize the reward data output by the reward model while controlling the magnitude of the policy model parameter changes during adjustment, preventing drastic changes in the policy parameters from causing instability or inefficiency in the learning process.

[0167] Based on this, when determining the benefit value using the first reward data and the similarity in this embodiment, the benefit value can be calculated using the following formula (3).

[0168]

[0169] Among them, J φ Used to express the return value; R θ It can be used to represent the reward data output by the reward model with parameter θ, and here it can be used to represent the first reward data; φ Used to represent the predicted action probability distribution output by the policy model when the parameter is φ; Used to represent the approximate distribution of preset actions output by the policy model when the parameter is φ0; It is used to express the similarity between the predicted action probability distribution and the preset action approximate distribution. In this embodiment, KL divergence is used as a regularization term to limit the degree of change of the predicted action probability distribution relative to the preset action probability distribution; β is a hyperparameter used to adjust the R θ and The proportional relationship between these two parts.

[0170] In this embodiment, when J φ When it is equal to or greater than the second preset threshold, it can be considered that the policy model has been trained.

[0171] The above method can be used to obtain a trained strategy model, which can be used to generate anthropomorphic behavioral strategies for each game agent in the camp in a team combat game scenario.

[0172] However, this approach only teaches the reward model how to make the state characteristics of the game agent resemble those of the player character, but does not optimize the strength of the reward model. This makes it difficult to train the game agent's behavioral strategies to match the gaming needs of high-level players. The strength of a reward model generally refers to how well the model fits the data and its predictive capabilities. A reward model is considered strong if it can accurately capture patterns and regularities in state characteristics and has good generalization capabilities for unknown data.

[0173] To address these shortcomings, this embodiment introduces a game environment strength reward to assist in optimizing the strategy model to meet the needs of high-level players. The win / loss information of a game match can be modeled as the game environment's strength reward. If the faction to which the game agent belongs wins the game match, the strength reward is 1; if it loses, the strength reward is -1.

[0174] Based on this, the “training the reward model to be trained according to the first reward data” in the above step S140 can also be specifically implemented through the following steps S143 to S144.

[0175] Step S143: Obtain the game result data of the current game in which the game agent participates in the game.

[0176] Step S144: Train the strategy model to be trained based on the first reward data and the game result data.

[0177] The game result data for the current game in which the agent is participating is used to represent the win / loss information of the game mentioned above. Taking the win / loss data of the game in which the agent is participating into account when training the strategy model can further improve the strategy model's ability to consider game win / loss outcomes when generating behavioral strategies for the agent, thereby enhancing the strategy model's predictive capabilities and optimizing its strength.

[0178] In a specific embodiment, the above step S144 can be specifically carried out through the following steps S1441 to S143.

[0179] Step S1441: Determine the strength reward data for the game agent based on the game result data.

[0180] Step S1442: Determine target reward data for the game agent based on the strength reward data and the first reward data.

[0181] Step S1443: The policy model to be trained is processed according to the target reward data.

[0182] The aforementioned Strength Reward data is used to assess the value of game match results. If the match result is a win, the Strength Reward data is 1; if the match result is a negative, the Strength Reward data is -1.

[0183] The target reward data is the reward data obtained by combining the first reward data output by the reward model with the intensity reward data. Specifically, the intensity reward data and the first reward data can be mixed according to the first preset weight coefficient to obtain the target reward data for the game agent. Specifically, the target reward data can be calculated using the following formula (4).

[0184] R final =Rθ +αR env (4)

[0185] Among them, R final Used to represent target reward data, R θ Used to represent the first reward data, R env It is used to represent the strength reward data, and α is used to represent the first preset weight coefficient.

[0186] After obtaining the target reward data, the policy model to be trained can be trained based on the target reward data. That is, the parameters of the policy model to be trained can be adjusted based on the target reward data to obtain a trained policy model. In this embodiment, the process of adjusting the parameters of the policy model to be trained based on the target reward data can refer to the process of adjusting the policy model to be trained based on the first reward data.

[0187] That is, in this embodiment, the profit value of the profit function in the strategy model is calculated by calculating the target reward data and the similarity, as shown in formula (5).

[0188]

[0189] The meaning of each parameter in the above formula (5) can be referred to the previous formula and will not be repeated here.

[0190] Next, in response to the defect in the related art that when the game version is updated, only the reward model and the strategy model can be retrained, and the training process takes too long, resulting in a long time for the game to be launched, the game strategy model generation method provided in this embodiment also includes a method for fine-tuning the reward model and the strategy model according to the game version update, so as to improve the training efficiency of the reward model and the strategy model, and avoid the problem of low efficiency caused by only retraining in the related art.

[0191] Assuming that the game data in the above step S110 is the first version data, the method for generating the strategy model provided in the embodiment of the present application may further include the following steps S150 to S160.

[0192] Step S150: When it is detected that the game data is updated to the second version data, the first version data and the second version data are mixed according to a second preset weight to obtain target game data.

[0193] Step S160: According to the target game data, the parameters of the trained reward model and the parameters of the trained strategy model are adjusted respectively to obtain an adjusted reward model and an adjusted strategy model.

[0194] The above detection that the game data is updated to the second version data can be understood as detection of a game version update.

[0195] like Figure 8 As shown, the second version data and the first version data can be mixed according to the second preset weight to obtain target game data, and then the reward model and the strategy model can be adjusted based on the target game data.

[0196] Regarding step S160, the parameters of the trained reward model and the parameters of the trained policy model can be fine-tuned using a low-rank adaptation (LoRA) model. The LoRA model is an algorithm for fine-tuning large pre-trained models. The parameters of the trained reward model or the parameters of the trained policy model can be adjusted by training only the low-rank matrix, thereby performing parameter adjustments without completely retraining the entire trained reward model or the trained policy model.

[0197] like Figure 9 As shown, Figure 9 A schematic diagram of the training structure of the LoRA model is provided. The LoRA model can be tuned by fixing the weights of the pre-trained model (the trained strategy model or the trained reward model) and then inputting the state features of the game agent in the target game data into the LoRA model to optimize the dimensionality reduction matrix A and the dimensionality increase matrix B to achieve the trained strategy model or the trained reward model. This optimization scheme can effectively reduce the number of adjustable parameters of the pre-trained model while ensuring the training effect. Model fine-tuning of the LoRA model is not specifically introduced in this embodiment.

[0198] So far, the strategy generation method provided in this embodiment has been explained. Since the above-mentioned generation process is to train the strategy model based on the trained reward model after the reward model is trained, the strategy model can be generated based on the optimal and more instructive reward data obtained from the trained reward model, rather than using less than ideal reward data to train the strategy model in the related technology. This makes the training process of the strategy model more stable, and generating the strategy model through more instructive reward data helps the game agent to understand and learn the desired behavior strategy more quickly when generating the strategy model, thereby achieving a better anthropomorphic effect.

[0199] In addition, during the generation of the above-mentioned strategy model, the training process of the reward model dynamically adjusts the network parameters of the reward model based on the purpose of enabling the game agent to simulate the state characteristics of the player character, which can reduce the difficulty of designing the reward model in related technologies. Moreover, through the feedback of the state characteristics of the human player on the overall behavior of the game agent, more accurate and adaptable reward data can be provided for training the strategy model.

[0200] Based on the above-mentioned method for generating a game strategy model provided in the embodiment of the present application, the embodiment of the present application also provides a method for generating a strategy of a game agent. In this embodiment, the method for generating a game strategy model provided in any of the above-mentioned embodiments can be used to obtain a trained game strategy model to generate a behavior strategy of a game agent. Figure 10 As shown, the strategy generation method of the game agent may specifically include the following steps S210 to S220.

[0201] Step S210: Obtain the game status data of the game agent in the game.

[0202] The game status data can be understood as the attribute information of the game agent in the game environment, and the attribute information may include but is not limited to position, speed, blood volume, life value, energy value, number of surrounding teammates or enemies, game terrain, game gain and other status data.

[0203] Step S220: Input the game state data of the game agent into the trained strategy model generated in steps S110 to S140 to obtain a behavior strategy generated by the trained strategy model according to the game state data of the game agent.

[0204] The strategy generation method of the game agent provided in this application may also include the following step S230.

[0205] Step S230: Control the game agent to perform actions according to the behavior strategy.

[0206] The behavioral strategies generated by the trained strategy model have a high degree of anthropomorphism, which can make the game actions of the game agent in the game environment more natural, accurately understand the combat intentions of the player characters in the camp, and then accurately cooperate with the player characters to carry out team combat, thereby improving the player's gaming experience.

[0207] Corresponding to the game strategy model generation method provided in the embodiment of the present application, the embodiment of the present application also provides a game strategy model generation device 300, such as Figure 11 As shown, the apparatus 300 includes: an acquisition unit 301, a training unit 302, and a processing unit 303;

[0208] The acquisition unit 301 is used to acquire a preset data set, which includes a first state feature sequence of a game agent and a second state feature sequence of a player character in the game;

[0209] The training unit 302 is configured to train the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is met, thereby obtaining a trained reward model;

[0210] The processing unit 303 is configured to input the target state characteristics of the game agent in the game into the trained reward model to obtain first reward data corresponding to the game agent;

[0211] The training unit 302 is further configured to train the strategy model to be trained according to the first reward data.

[0212] Optionally, the training unit 302 is specifically used to input the first state feature sequence and the second state feature sequence into the reward model to be trained, obtain the second reward data corresponding to the game agent, and the third reward data corresponding to the player character; according to the second reward data and the third reward data, adjust the parameters of the reward model to be trained until the preset training end conditions are reached, and obtain the trained reward model.

[0213] Optionally, the first state feature sequence is a sequence composed of state features of the game agent at at least one time node;

[0214] The training unit 302 is specifically used to determine the reward values ​​corresponding to the state characteristics of each time node in the first state feature sequence of the game agent through the reward model to be trained; and determine the sum of the reward values ​​corresponding to each time node as the second reward data fed back by the game environment to the game agent.

[0215] Optionally, the game is a team combat game, the game includes at least one camp, the camp includes multiple virtual characters, and the multiple virtual characters include at least one of the game agents;

[0216] The training unit 302 is specifically used to determine the first state feature corresponding to the i-th time node in the first state feature sequence of the game agent; determine the target team state feature corresponding to the i-th time node of the camp to which the game agent belongs based on the first state feature corresponding to the i-th time node; encode the target team state feature corresponding to the i-th time node to obtain the reward value fed back by the game environment to the game agent at the i-th time node; wherein, i traverses 1 to N, N is an integer, and N is the number of time nodes included in the first state feature sequence.

[0217] Optionally, the training unit 302 is specifically used to determine the first target feature corresponding to the game agent in the first state feature corresponding to the i-th time node and the second target feature corresponding to each virtual character in the camp except the game agent; wherein the first target feature is used to characterize the game state feature corresponding to the game agent at the i-th time node, and the second target feature is used to characterize the game state feature corresponding to the virtual character at the i-th time node; based on the first target feature and the second target feature, the target team state feature corresponding to the camp to which the game agent belongs at the i-th time node is determined.

[0218] Optionally, the training unit 302 is specifically used to determine the correlation data between the first target feature and the second target feature, and determine the target team status feature corresponding to the camp to which the game agent belongs at the i-th time node based on the correlation data.

[0219] Optionally, the acquiring unit 301 is further configured to acquire a historical team status feature corresponding to the camp to which the game agent belongs at the i-1th time node;

[0220] The training unit 302 is also used to determine the historical team status characteristics corresponding to the camp to which the game agent belongs at the i-th time node based on the historical team status characteristics and the target team status characteristics; encode the historical team status characteristics corresponding to the i-th time node to obtain the reward value that the game environment feeds back to the game agent at the i-th time node.

[0221] Optionally, the training unit 302 is specifically used to determine the loss value of the loss function corresponding to the reward model to be trained based on the second reward data and the third reward data; according to the principle that the loss value is less than or equal to the first preset threshold, adjust the parameters of the reward model to be trained to obtain the trained reward model.

[0222] Optionally, the training unit 302 is further used to input the target state characteristics of the game agent in the game into the strategy model to be trained to obtain a predicted action probability distribution; and train the strategy model to be trained based on the first reward data and the predicted action probability distribution.

[0223] Optionally, the training unit 302 is specifically used to determine the similarity between the predicted action probability distribution and the preset action probability distribution for the game agent; determine the benefit value of the benefit function corresponding to the strategy model to be trained based on the first reward data and the similarity, and adjust the parameters of the strategy model to be trained according to the principle that the benefit value is equal to or greater than a second preset threshold to obtain a trained strategy model.

[0224] Optionally, the acquisition unit 301 is further configured to acquire game result data of a current game in which the game agent participates in the game;

[0225] The training unit 302 is specifically configured to train the strategy model to be trained based on the first reward data and the game result data.

[0226] Optionally, the training unit 302 is specifically used to determine the strength reward data for the game agent based on the game result data; determine the target reward data for the game agent based on the strength reward data and the first reward data; and train the strategy model to be trained based on the target reward data.

[0227] Optionally, the training unit 302 is further configured to mix the intensity reward data and the first reward data according to a first preset weight coefficient to obtain target reward data for the game agent.

[0228] Optionally, the game data is first version data;

[0229] The processing unit 303 is also used to, when it is detected that the game data is updated to the second version data, mix the first version data and the second version data according to a second preset weight to obtain target game data; and adjust the trained reward model and the trained strategy model according to the target game data to obtain an adjusted reward model and an adjusted strategy model.

[0230] Corresponding to the method for generating a behavior strategy of a game agent provided in the embodiment of the present application, the embodiment of the present application also provides a device 400 for generating a behavior strategy of a game agent, such as Figure 12 As shown, the device 400 includes: an acquisition unit 401 and a generation unit 402.

[0231] An acquisition unit 401 is configured to acquire game state data of the game agent in the game;

[0232] The generation unit 402 is used to input the game status data of the game agent into the trained strategy model obtained by any of the game strategy model generation methods, and obtain the behavioral strategy generated by the trained strategy model based on the game status data of the game agent.

[0233] Optionally, the device 400 further includes a control unit 403;

[0234] The control unit 403 is used to control the game agent to perform actions according to the behavior strategy.

[0235] The embodiment of the present application also provides an electronic device for implementing a method for generating a game strategy model or a method for generating a behavior strategy of a game agent, such as Figure 13 As shown, the electronic device includes: a processor 501; and a memory 502, which is used to store a program of a game strategy model generation method or a game agent behavior strategy generation method. After the device is powered on and the program of the game strategy model generation method is run by the processor, the following steps are performed:

[0236] Obtaining a preset data set, the preset data set including a first state feature sequence of a game agent and a second state feature sequence of a player character in the game;

[0237] Training the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is met, thereby obtaining a trained reward model;

[0238] Inputting the target state feature of the game agent in the game into the trained reward model to obtain first reward data corresponding to the game agent;

[0239] The policy model to be trained is trained according to the first reward data.

[0240] After the device is powered on and the program of the method for generating the behavior strategy of the game agent is run through the processor, the following steps are performed:

[0241] Obtaining game status data of the game agent in the game;

[0242] The game state data of the game agent is input into a trained strategy model obtained by any one of the game strategy model generation methods, and a behavior strategy generated by the trained strategy model according to the game state data of the game agent is obtained.

[0243] The present application also provides a computer-readable storage medium storing a program for a method for generating a game strategy model or a method for generating a behavior strategy of a game agent. The program is executed by a processor to perform the following steps:

[0244] Obtaining a preset data set, the preset data set including a first state feature sequence of a game agent and a second state feature sequence of a player character in the game;

[0245] Training the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is met, thereby obtaining a trained reward model;

[0246] Inputting the target state feature of the game agent in the game into the trained reward model to obtain first reward data corresponding to the game agent;

[0247] The policy model to be trained is trained according to the first reward data.

[0248] Alternatively, perform the following steps:

[0249] Obtaining game status data of the game agent in the game;

[0250] The game state data of the game agent is input into a trained strategy model obtained by any one of the game strategy model generation methods, and a behavior strategy generated by the trained strategy model according to the game state data of the game agent is obtained.

[0251] It should be noted that for the detailed description of the device, electronic device and computer-readable storage medium provided in the embodiments of the present application, reference can be made to the relevant description of the game strategy model generation method or the game agent behavior strategy generation embodiment provided in the embodiments of the present application, and no further details will be given here.

[0252] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0253] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0254] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0255] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable operations, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0256] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0257] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for generating a game strategy model, characterized in that: The method comprises: Acquire a preset data set, wherein the preset data set includes a first state feature sequence of a game agent and a second state feature sequence of a player character in the game; Training the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is reached, thereby obtaining a trained reward model; Inputting the target state feature of the game agent in the game into the trained reward model to obtain first reward data corresponding to the game agent; The policy model to be trained is trained according to the first reward data.

2. The method according to claim 1, characterized in that The reward model to be trained is trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is reached to obtain a trained reward model, including: Inputting the first state feature sequence and the second state feature sequence into a reward model to be trained to obtain second reward data corresponding to the game agent and third reward data corresponding to the player character; According to the second reward data and the third reward data, the parameters of the reward model to be trained are adjusted until a preset training end condition is reached, thereby obtaining a trained reward model.

3. The method according to claim 2, characterized in that The first state feature sequence is a sequence composed of state features of the game agent at at least one time node; The step of inputting the first state feature sequence into the reward model to be trained to obtain second reward data corresponding to the game agent includes: Determining, by means of the reward model to be trained, the reward value corresponding to the state feature of each of the time nodes in the first state feature sequence of the game agent; The sum of the reward values ​​corresponding to each of the time nodes is determined as the second reward data fed back by the game environment to the game agent.

4. The method according to claim 3, characterized in that The game is a team combat game, including at least one camp, the camp including multiple virtual characters, and the multiple virtual characters including at least one of the game agents; Determining the reward value corresponding to each state feature of the time node in the first state feature sequence of the game agent includes: Determining a first state feature corresponding to an i-th time node in the first state feature sequence of the game agent; Determine, based on the first state feature corresponding to the i-th time node, the target team state feature of the camp to which the game agent belongs at the i-th time node; Encode the target team state feature corresponding to the i-th time node to obtain the reward value fed back by the game environment to the game agent at the i-th time node; Here, i traverses 1 to N, N is an integer, and N is the number of time nodes included in the first state feature sequence.

5. The method according to claim 4, characterized in that The determining, based on the first state feature corresponding to the i-th time node, the target team state feature of the camp to which the game agent belongs corresponding to the i-th time node includes: Determine a first target feature corresponding to the game agent in the first state feature corresponding to the i-th time node and a second target feature corresponding to each virtual character in the camp other than the game agent; wherein the first target feature is used to characterize the game state feature corresponding to the game agent at the i-th time node, and the second target feature is used to characterize the game state feature corresponding to the virtual character at the i-th time node; According to the first target feature and the second target feature, the target team state feature corresponding to the camp to which the game agent belongs at the i-th time node is determined.

6. The method according to claim 5, characterized in that The determining, based on the first target feature and the second target feature, the target team status feature corresponding to the camp to which the game agent belongs at the i-th time node includes: Determine the correlation data between the first target feature and the second target feature, and determine the target team status feature corresponding to the camp to which the game agent belongs at the i-th time node based on the correlation data.

7. The method according to any one of claims 4 to 6, characterized in that After determining the target team status feature corresponding to the camp to which the game agent belongs at the i-th time node, and before encoding the target team status feature corresponding to the i-th time node to obtain the reward value fed back by the game environment to the game agent at the i-th time node, the method further includes: Obtain the historical team status characteristics of the camp to which the game agent belongs at the i-1th time node; Determine, based on the historical team status characteristics and the target team status characteristics, the historical team status characteristics corresponding to the camp to which the game agent belongs at the i-th time node; The encoding of the target team state feature corresponding to the i-th time node to obtain the reward value fed back by the game environment to the game agent at the i-th time node includes: The historical team status features corresponding to the i-th time node are encoded to obtain the reward value fed back by the game environment to the game agent at the i-th time node.

8. The method according to claim 2, characterized in that The preset training end condition includes that the loss value of the reward model to be trained is less than or equal to a first preset threshold; the parameters of the reward model to be trained are adjusted according to the second reward data and the third reward data until the preset training end condition is met, thereby obtaining a trained reward model, including: Determining a loss value of a loss function corresponding to the reward model to be trained based on the second reward data and the third reward data; According to the principle that the loss value is less than or equal to the first preset threshold, the parameters of the reward model to be trained are adjusted to obtain a trained reward model.

9. The method according to claim 2, characterized in that The training of the strategy model to be trained according to the first reward data includes: Inputting the target state characteristics of the game agent in the game into the strategy model to be trained to obtain a predicted action probability distribution; The policy model to be trained is trained according to the first reward data and the predicted action probability distribution.

10. The method according to claim 9, characterized in that The training of the strategy model to be trained according to the first reward data and the predicted action probability distribution includes: Determining a similarity between the predicted action probability distribution and a preset action probability distribution for the gaming agent; Determine the benefit value of the benefit function corresponding to the strategy model to be trained based on the first reward data and the similarity, and adjust the parameters of the strategy model to be trained according to the principle that the benefit value is greater than or equal to a second preset threshold.

11. The method according to claim 1, wherein The training of the strategy model to be trained according to the first reward data includes: Obtaining game result data of a current game in which the game agent participates in the game; The strategy model to be trained is trained according to the first reward data and the game result data.

12. The method according to claim 11, characterized in that The step of training the strategy model to be trained based on the first reward data and the game result data includes: Determining strength reward data for the game agent based on the game result data; Determining target reward data for the game agent based on the strength reward data and the first reward data; The strategy model to be trained is trained according to the target reward data.

13. The method according to claim 12, characterized in that The determining of target reward data for the game agent based on the strength reward data and the first reward data includes: The intensity reward data and the first reward data are mixed according to a first preset weight coefficient to obtain target reward data for the game agent.

14. The method according to claim 1, wherein The game data is first version data; the method further includes: When it is detected that the game data is updated to the second version data, the first version data and the second version data are mixed according to a second preset weight to obtain target game data; According to the target game data, the trained reward model and the trained strategy model are adjusted respectively to obtain an adjusted reward model and an adjusted strategy model.

15. A method for generating a behavioral strategy for a game agent, characterized in that: The method comprises: Obtaining game status data of the game agent in the game; The game state data of the game agent is input into a trained strategy model obtained by the game strategy model generation method described in any one of claims 1 to 14, and a behavioral strategy generated by the trained strategy model according to the game state data of the game agent is obtained.

16. The method according to claim 15, characterized in that The method further comprises: The game agent is controlled to perform actions according to the behavior strategy.

17. A device for generating a game strategy model, characterized in that: The device comprises: an acquisition unit, a training unit and a processing unit; The acquisition unit is used to acquire a first state feature sequence of the game agent and a second state feature sequence of the player character in the game data of the game; The training unit is configured to train the reward model to be trained according to the first state feature sequence and the second state feature sequence until a preset training end condition is met, thereby obtaining a trained reward model; The processing unit is configured to input the first state feature sequence into the trained reward model to obtain first reward data fed back by the game environment to the game agent; The training unit is further configured to train the strategy model to be trained according to the first reward data.

18. A device for generating a behavior strategy of a game agent, characterized in that: The device includes: an acquisition unit and a generation unit; An acquisition unit, configured to acquire game state data of the game agent in the game; A generation unit is used to input the game status data of the game agent into a trained strategy model obtained by the game strategy model generation method described in any one of claims 1 to 14, and obtain a behavioral strategy generated by the trained strategy model based on the game status data of the game agent.

19. An electronic device, characterized in that: include: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 16 is executed.

20. A computer-readable storage medium, characterized in that A data processing program is stored, and the program is run by a processor to execute the method according to any one of claims 1 to 16.

Images