Method, device, medium and computer program product for training a game strategy model

Abstract

The application relates to the technical field of artificial intelligence, and provides a method and device for training a battle strategy model of a multi-player battle game, a computer device, a storage medium and a computer program product. The method comprises the following steps: obtaining each historical state from battle data corresponding to a virtual object lineup for each virtual object lineup, determining a game target corresponding to each historical state, determining a battle strategy type, training a game target prediction model based on the historical state, the battle strategy type and the reward data, inputting a response state corresponding to each battle action of each virtual object lineup based on an initial battle strategy model and a battle strategy type corresponding to the corresponding virtual object lineup into the game target prediction model to obtain a predicted game target, and training the initial battle strategy model based on reward data corresponding to each predicted game target to obtain a battle strategy model, so that the confrontation ability of the virtual object lineup can be effectively improved.

Description

[0001] This application claims priority to Chinese Patent Application No. 2021110771354, filed on September 13, 2021, entitled “Training Method, Apparatus, Medium and Computer Program Product for Battle Strategy Model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for training a battle strategy model for a multiplayer battle game. Background Technology

[0003] With the development of computer technology, computer games and other human-computer interaction applications, such as Multiplayer Online Battle Arena (MOBA) games, have become an increasingly popular form of entertainment. In MOBA games, players are typically divided into multiple opposing factions, controlling virtual objects to compete and achieve game objectives to win. In human-computer battles or game automation, a computer, such as game AI (Artificial Intelligence), automatically controls the computer player or the corresponding virtual object to engage in combat. The game AI needs to learn strategies and tactics for gameplay.

[0004] For multiplayer online tactical battle royale games, reinforcement learning is usually used to train game AI. In the reinforcement learning process, reward data is used to guide the learning of game AI. However, most of the current reward data adopts a dense reward design defined by humans. The reward design is the same for each game AI and also for each lineup. This results in different lineups having the same values. Any lineup combination presents the same strategy and gameplay, resulting in low competitive ability of lineup combinations composed of multiple virtual objects in the game. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, apparatus, computer equipment, storage medium, and computer program product for training a battle strategy model in a multiplayer game that can improve the competitive ability of lineup combinations composed of multiple virtual objects in the game, in order to address the above-mentioned technical problems.

[0006] A method for training a battle strategy model in a multiplayer game, the method comprising:

[0007] For each virtual object lineup, obtain each historical state from the battle data corresponding to the virtual object lineup, determine the game objective corresponding to each of the historical states for the virtual object lineup, and obtain the battle strategy type of the virtual object lineup.

[0008] Using the historical state and the battle strategy type as input data, and the game objective as the label of the input data, a game objective prediction model is trained.

[0009] For each battle action of each virtual object lineup obtained based on the initial battle strategy model, the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0010] Based on the reward data corresponding to each of the predicted game objectives, the initial battle strategy model is trained to obtain the battle strategy model.

[0011] A training device for a battle strategy model in a multiplayer game, the device comprising:

[0012] The data acquisition module is used to acquire historical states from the battle data corresponding to each virtual object lineup, determine the game objective corresponding to each of the historical states of the virtual object lineup, and acquire the battle strategy type of the virtual object lineup for each virtual object lineup.

[0013] The game objective prediction model training module is used to train a game objective prediction model by taking the historical state and the battle strategy type as input data and the game objective as the label of the input data.

[0014] The game objective prediction module is used to input the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup into the game objective prediction model for each battle action of each virtual object lineup obtained based on the initial battle strategy model, so as to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0015] The battle strategy model training module is used to train the initial battle strategy model based on the reward data corresponding to each of the predicted game objectives, so as to obtain the battle strategy model.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0017] For each virtual object lineup, obtain each historical state from the battle data corresponding to the virtual object lineup, determine the game objective corresponding to each of the historical states for the virtual object lineup, and obtain the battle strategy type of the virtual object lineup.

[0018] Using the historical state and the battle strategy type as input data, and the game objective as the label of the input data, a game objective prediction model is trained.

[0019] For each battle action of each virtual object lineup obtained based on the initial battle strategy model, the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0020] Based on the reward data corresponding to each of the predicted game objectives, the initial battle strategy model is trained to obtain the battle strategy model.

[0021] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0022] For each virtual object lineup, obtain each historical state from the battle data corresponding to the virtual object lineup, determine the game objective corresponding to each of the historical states for the virtual object lineup, and obtain the battle strategy type of the virtual object lineup.

[0023] Using the historical state and the battle strategy type as input data, and the game objective as the label of the input data, a game objective prediction model is trained.

[0024] For each battle action of each virtual object lineup obtained based on the initial battle strategy model, the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0025] Based on the reward data corresponding to each of the predicted game objectives, the initial battle strategy model is trained to obtain the battle strategy model.

[0026] A computer program product includes a computer program, characterized in that, when the computer program is executed by a processor, it performs the following steps:

[0027] For each virtual object lineup, obtain each historical state from the battle data corresponding to the virtual object lineup, determine the game objective corresponding to each of the historical states for the virtual object lineup, and obtain the battle strategy type of the virtual object lineup.

[0028] Using the historical state and the battle strategy type as input data, and the game objective as the label of the input data, a game objective prediction model is trained.

[0029] For each battle action of each virtual object lineup obtained based on the initial battle strategy model, the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0030] Based on the reward data corresponding to each of the predicted game objectives, the initial battle strategy model is trained to obtain the battle strategy model.

[0031] The aforementioned training method, apparatus, computer equipment, storage medium, and computer program product for the combat strategy model of multiplayer games, by acquiring historical states from the combat data corresponding to each virtual object lineup, determining the game objective corresponding to each historical state of the virtual object lineup, and acquiring the combat strategy type of the virtual object lineup, and using the historical states and combat strategy types as input data, and the game objective as the label of the input data, a game objective prediction model is trained. This game objective prediction model can guide the learning of the initial combat strategy model. For each combat action of each virtual object lineup obtained based on the initial combat strategy model, the response state corresponding to the corresponding combat action and the combat strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup. Based on the reward data corresponding to each predicted game objective, the initial combat strategy model is trained to obtain the combat strategy model. This allows the combat strategy model to learn different strategies for different virtual object lineups, achieving strategy diversity, thereby improving the combat capability of lineups composed of multiple virtual objects in the game. Attached Figure Description

[0032] Figure 1 This is an application environment diagram of a multiplayer game strategy model training method in one embodiment.

[0033] Figure 2 This is a flowchart illustrating a method for training a battle strategy model in a multiplayer game, as shown in one embodiment.

[0034] Figure 3This is a flowchart illustrating a method for training a battle strategy model in a multiplayer game, as described in another embodiment.

[0035] Figure 4 This is a flowchart illustrating a method for training a battle strategy model in a multiplayer game, as shown in another embodiment.

[0036] Figure 5 This is a flowchart illustrating a method for training a battle strategy model in a multiplayer game, as shown in one embodiment.

[0037] Figure 6 This is a data processing flowchart for a game objective prediction model and a battle strategy model in one embodiment;

[0038] Figure 7 This is a schematic diagram illustrating the training process of a game target prediction model in one embodiment;

[0039] Figure 8 This is a schematic diagram of the data processing flow of a battle strategy model in one embodiment;

[0040] Figure 9 This is a schematic diagram illustrating the application process of the trained game objective prediction model and battle strategy model in one embodiment.

[0041] Figure 10 This is a structural block diagram of a multiplayer game strategy model training device in one embodiment.

[0042] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0046] Machine learning is a multidisciplinary field that studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning (RL), transfer learning, inductive learning, and instructional learning.

[0047] Reinforcement learning, also known as reward learning, evaluation learning, or reinforcement learning, is a paradigm and methodology in machine learning used to describe and solve problems where an agent learns strategies to maximize rewards or achieve specific goals during interactions with its environment. Reinforcement learning involves the agent learning through trial and error, using rewards gained from interactions with the environment to guide behavior, with the goal of maximizing the agent's reward. Reinforcement learning differs from supervised learning in connectionist learning, primarily in the reinforcement signal. In reinforcement learning, the reinforcement signal provided by the environment is an evaluation of the quality of the action (usually a scalar signal), rather than telling the reinforcement learning system (RLS) how to produce the correct action. Because the external environment provides limited information, RLS must learn through its own experience. In this way, RLS acquires knowledge in the action-evaluation environment and improves its action plans to adapt to the environment.

[0048] A virtual scene is a virtual environment displayed (or provided) by an application when it runs on a terminal. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application embodiment does not limit the dimension of the virtual scene. For example, a virtual scene includes a sky, land, ocean, etc., where the land includes environmental elements such as deserts and cities, and the end user can control virtual objects to move within the virtual scene. Optionally, the virtual scene can also be used for virtual scene battles between at least two virtual objects, and the virtual scene has virtual resources available for use by at least two virtual objects.

[0049] A virtual object refers to an active object in a virtual scene. This active object can be a virtual character, virtual animal, anime character, etc., such as a person, animal, plant, oil drum, wall, or stone displayed in the virtual scene. The virtual object can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the space within the virtual scene. Optionally, when the virtual scene is a three-dimensional virtual scene, the virtual object can be a three-dimensional model. This three-dimensional model can be a three-dimensional character constructed based on three-dimensional human skeleton technology. The same virtual object can display different appearances by wearing different skins. In some embodiments, virtual objects can also be implemented using 2.5D or 2D models; this application does not limit this.

[0050] The multiplayer game strategy model training method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 and server 104 can be connected directly or indirectly via wired or wireless communication, which is not limited herein. Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart gaming terminal, etc., but is not limited to these. Terminal 102 has an application that supports virtual scenes installed and running. This application can be any of the following: online tactical competitive game, real-time strategy game, virtual reality application, 3D map program, shooting game, or multiplayer survival game. In a specific application, terminal 102 is a user-operated terminal. The user uses terminal 102 to operate a controlled virtual object located in the virtual scene, and this activity includes, but is not limited to, at least one of the following: adjusting body posture, crawling, walking, running, riding, jumping, driving, picking up, shooting, attacking, throwing, and releasing skills. Illustratively, the virtual object is a virtual character, such as a realistic or anime character.

[0051] Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 104 is used to provide backend services for applications that support virtual scenarios.

[0052] In a specific application, the virtual object controlled by terminal 102 (hereinafter referred to as the controlled virtual object) and the virtual object controlled by server 104 (hereinafter referred to as the AI ​​object) are in the same virtual scene. At this time, the controlled virtual object can interact with the AI ​​object in the virtual scene. In some embodiments, the controlled virtual object and the AI ​​object can be in an adversarial relationship. For example, the controlled virtual object and the AI ​​object can belong to different teams and organizations. The adversarial virtual objects can engage in combative interaction by releasing skills to each other.

[0053] In one embodiment, such as Figure 2 As shown, a method for training a battle strategy model in a multiplayer game is provided, and this method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0054] Step 202: For each virtual object lineup, obtain each historical state from the battle data corresponding to the virtual object lineup, determine the game objective corresponding to each virtual object lineup in each historical state, and obtain the battle strategy type of the virtual object lineup.

[0055] In a competitive virtual environment, a virtual object lineup refers to the combination and configuration of multiple virtual objects within the same faction, forming a combat team. A virtual object lineup includes the combination of virtual objects and the configuration data of each virtual object. This configuration data is related to the game's combat style; for example, it can include in-game mission information, identity information, and position information. For instance, in a MOBA game, the configuration data of virtual objects can be determined based on the development position of a hero. The same hero can correspond to different development positions in different combinations, and different heroes can correspond to the same development position in different combinations. For example, hero A might be a "top laner" in combination 1 but a "support" in combination 2. Using the combination of virtual objects and the configuration data of each virtual object allows for a more accurate description of the virtual object lineup.

[0056] The virtual scene can be a virtual scene in a MOBA game, including various virtual resources such as controllable virtual objects and non-user characters set up for interaction within the virtual scene, such as monsters, turrets, and minions. Virtual objects are user characters controlled through client-side operations, or are artificial intelligence trained and set up for virtual scene battles. Optionally, the virtual object is a virtual character engaging in adversarial interaction within the virtual scene. The number of virtual objects participating in the interaction in the virtual scene can be preset or dynamically determined based on the number of clients joining the interaction.

[0057] MOBA (Multiplayer Online Battle Arena) games are games that provide several checkpoints in a virtual environment, where users from different factions control virtual characters to battle against each other, capturing or destroying checkpoints belonging to the opposing faction. For example, a MOBA game can divide users into at least two opposing factions, with different virtual teams belonging to each faction occupying their respective map areas, competing against each other with a specific victory condition. This victory condition includes, but is not limited to: capturing or destroying checkpoints, eliminating virtual characters from the opposing faction, surviving within a specified time and scenario, acquiring a resource, or exceeding the opponent's score within a specified time. For example, a MOBA game can divide users into two opposing factions, distributing their controlled virtual characters across a virtual environment to compete, with the victory condition being the destruction or capture of all enemy checkpoints.

[0058] Historical state refers to the current location of various virtual resources within the virtual environment at any given moment during the battle. This includes the states of controllable virtual objects within the virtual environment and the states of non-user characters interacting in the virtual scene. Examples include the state of each virtual object in the virtual scene at a specified moment, including its location, kill count, death count, economic value, and battle time; and the states of monsters, turrets, and minions in the virtual scene at a specified moment.

[0059] A game objective refers to a goal achieved or a task completed during a game. Game objectives can correspond to the state at the moment the objective is achieved or the task is completed in the game's history, or they can directly use the historical state at the moment the objective is achieved or the task is completed as the game objective. Each game contains at least one game objective. Game objectives can be conditions for achieving final victory, such as capturing or destroying a stronghold, eliminating a virtual enemy, surviving within a specified timeframe, acquiring a resource, or surpassing the opponent's score within a specified time. Game objectives can also be a set of conditions for partial victory and conditions for achieving final victory, such as killing jungle monsters, clearing minion waves, or destroying towers. Game objectives can be a set of objectives constructed based on the game type. During each game, the opposing virtual enemy team will complete at least some of the game objectives in this set. Generally, game objectives differ across time periods; for example, the objective in the first time period might be killing jungle monsters, while the objective in the second time period might be destroying towers. Game objectives remain largely unchanged over a period of time, ensuring that game objectives are sparse data. The historical states presented in order to achieve a certain game objective all correspond to that game objective.

[0060] The combat strategy type characterizes the type of strategy employed throughout the game's combat process. Combat strategy types are distinguished by grouping strategies with the same tactics together. Specifically, combat strategy types can be categorized based on expert experience or artificial intelligence algorithms such as cluster analysis. By determining the combat strategy types corresponding to different virtual object lineups, the feature data of this dimension can be fully considered during the training of the game objective prediction model, ensuring a strong correlation between the trained game objective prediction model and the combat strategy type.

[0061] Specifically, for each virtual object lineup, the server needs to obtain the corresponding historical state, the game objective corresponding to the historical state, and the battle strategy type. The processing is the same for each virtual object lineup, which requires obtaining each historical state from the battle data corresponding to the virtual object lineup, determining the game objective corresponding to each historical state, and obtaining the battle strategy type of the virtual object lineup.

[0062] Step 204: Using historical states and battle strategy types as input data, and game objectives as labels for input data, train a game objective prediction model.

[0063] In this context, "historical state" refers to the current state of various virtual resources within the virtual environment at any given moment during the battle. Each historical state can be represented by state features, which can be feature vectors, feature matrices, etc., without specific limitations. "Battle strategy type" represents the type of strategy employed throughout the game's battle process. Each historical state can be represented by strategy type features, which have the same data format as the state features, such as feature vectors, feature matrices, etc.

[0064] Specifically, the server takes each historical feature and each battle strategy type as a set of input data. The game objective corresponding to the historical feature data in each set of input data is used as the label for that set. Each set of labeled input data is then fed into the initial model for supervised learning, resulting in a game objective prediction model. Training ends when the training conditions are met, such as reaching a threshold for the number of training iterations or a threshold for the model's output accuracy. The game objective prediction model is trained based on existing historical states, battle strategy types, and game objectives. It can predict game objectives based on the input state and battle strategy type, outputting the game objective to be achieved under the given conditions.

[0065] Step 206: For each battle action of each virtual object lineup obtained based on the initial battle strategy model, input the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0066] The initial battle strategy model is based on input state data, and the output is the corresponding battle action performed by each virtual object in the virtual object lineup. Battle actions are the specific ways to operate virtual objects to engage in game combat, such as controlling virtual objects to move and releasing skills.

[0067] The response state is the overall state of the game environment after the game environment has changed in response to the combat actions of each virtual object in the virtual object lineup. The response state is related to both the specific combat action and the previous state. If the combat actions of the virtual objects in the virtual object lineup remain unchanged, the response state will differ depending on the previous state. Conversely, if the combat actions of the virtual objects in the virtual object lineup change, the response state will also change if the previous state is the same.

[0068] The correspondence between virtual object lineups and battle strategy types is fixed. The battle strategy type corresponding to a virtual object lineup can be directly found from the training data of the game objective prediction model based on the virtual object lineup.

[0069] Specifically, the initial battle strategy model in the server can output corresponding battle actions based on the input state and actual reward data of the virtual object lineup. For each output battle action, the game system will, based on a pre-designed reward signal and the matching relationship between the battle action and the reward signal, provide the actual reward data corresponding to that output battle action, determine the changed environment based on the battle action, and obtain the response state based on the changed environment and each virtual object in the virtual object lineup after executing the battle action. The actual reward data input to the initial battle strategy model is the result given by the game system based on the previous battle action.

[0070] Furthermore, the processing procedure for each combat action of each virtual object lineup is the same as described above and will not be repeated here. For each combat action of each virtual object lineup obtained based on the initial combat strategy model, the server takes the data combination consisting of the response state corresponding to the corresponding combat action and the combat strategy type corresponding to the corresponding virtual object lineup as input data, and inputs it into the game objective prediction model to predict the game objective. The game objective prediction model outputs the predicted game objective corresponding to each response state of each virtual object lineup.

[0071] Step 208: Based on the reward data corresponding to each predicted game objective, train the initial battle strategy model to obtain the battle strategy model.

[0072] The reward data serves as the data guiding the initial combat strategy model in its strategy learning. Unlike the actual reward data corresponding to manually designed reward signals, which are directly linked to the combat results of combat actions, the actual reward data corresponds one-to-one with the combat results of combat actions. When the combat results are the same, the corresponding actual reward data will also be the same. The reward data corresponding to each predicted game objective can be intrinsic reward data. While intrinsic reward data corresponds to the predicted game objective, the relationship between it and the combat results of combat actions is not one-to-one. At different times, the intrinsic reward data obtained based on the same combat result may be different, and the intrinsic reward data obtained based on different combat results may also be the same.

[0073] Specifically, the server adds the reward data corresponding to each predicted game objective as new data, combines it with the corresponding state data and actual reward data, and inputs it into the initial battle strategy model for reinforcement learning to obtain the battle strategy model. The biggest difference between reinforcement learning and supervised learning is that reinforcement learning does not have the pre-prepared labels of supervised learning. Reinforcement learning only has reward data, but this reward data is different from the labels in supervised learning. It is not given in advance, but rather given afterward, and each step of reinforcement learning is closely related to the time sequence. For example, the battle strategy model determines the next battle action of the virtual object lineup based on the transition state. The game system provides the actual reward data corresponding to this battle action, and the predicted game objective obtained based on the game objective prediction model also receives corresponding reward data.

[0074] In this embodiment, the server obtains historical states from the battle data corresponding to each virtual object lineup, determines the game objective corresponding to each historical state, and obtains the battle strategy type of the virtual object lineup. Using historical states and battle strategy types as input data, and game objectives as labels, a game objective prediction model is trained. This game objective prediction model can guide the learning of the initial battle strategy model. For each battle action of each virtual object lineup obtained based on the initial battle strategy model, the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup. Based on the reward data corresponding to each predicted game objective, the initial battle strategy model is trained to obtain a battle strategy model. This allows the battle strategy model to learn different strategies for different virtual object lineups, achieving strategy diversity and thereby improving the combat capability of lineups composed of multiple virtual objects in the game.

[0075] In one embodiment, a game objective prediction model is trained using historical states and battle strategy types as input data, and game objectives as labels for the input data. This includes: for each virtual object lineup, concatenating the state features corresponding to each historical state of the virtual object lineup with the strategy type features corresponding to the battle strategy type to obtain input feature data; using the game objective corresponding to each input feature data as the label for the corresponding input feature data; and inputting the input feature data and labels corresponding to each virtual object lineup into the initial prediction model for training to obtain the game objective prediction model.

[0076] Among them, state features are the data feature representation of historical states, and strategy type features are the data feature representation of battle strategy types. State features and strategy type features have the same data type, which can be feature vectors, feature matrices, etc. Feature concatenation refers to the process of connecting two or more features with the same data type according to certain rules to obtain a complete feature. For example, the state feature vector corresponding to a preset dimension of historical states and the strategy type feature vector corresponding to a preset dimension of battle strategy types are concatenated according to their corresponding dimensions to obtain the input feature vector. The initial prediction model is the state before the game objective prediction model is trained.

[0077] Specifically, for each virtual object lineup, the server concatenates the state features corresponding to each historical state of the virtual object lineup with the strategy type features corresponding to the battle strategy type to obtain multiple input feature data. For each input feature data, the game objective corresponding to the historical state of the corresponding input feature data is used as the label of the input feature data. The labeled input feature data corresponding to each virtual object lineup is input into the initial prediction model for training. When the training termination condition is met, such as the number of training times reaching the number threshold or the output accuracy of the model reaching the accuracy threshold, the training ends, and the trained game objective prediction model is obtained.

[0078] In this embodiment, the server concatenates the state features corresponding to each historical state with the strategy type features corresponding to the battle strategy type. This enables each input data to simultaneously possess features of both the corresponding state and the corresponding battle strategy type, enhancing the correlation between the state and the battle strategy type. Consequently, the trained game objective prediction model becomes strongly correlated with the battle strategy type, improving the accuracy of prediction results for game objectives with different virtual object lineups.

[0079] In one embodiment, such as Figure 3 As shown, for each virtual object lineup, historical states are obtained from the corresponding battle data of the virtual object lineup. The game objective corresponding to each historical state of the virtual object lineup is determined, and the battle strategy type of the virtual object lineup is obtained, including:

[0080] Step 302: For each virtual object lineup, retrieve the corresponding battle data from the lineup pool to which the virtual object lineup belongs.

[0081] Step 304: Extract the historical state of the virtual object lineup during the battle and the game objectives completed during the battle from the battle data.

[0082] Step 306: Based on the matching relationship between each game objective and each historical state, determine the game objective corresponding to each virtual object lineup in each historical state.

[0083] Step 308: Determine the battle strategy type of the virtual object lineup based on the correspondence between the lineup pool and the battle strategy type.

[0084] The lineup pool is a container used to store the battle data corresponding to virtual object lineups. A single lineup pool includes battle data corresponding to one or more virtual object lineups, and the battle strategy type and game objective are the same for all virtual object lineups within the same pool. Specifically, once the lineup pool corresponding to a virtual object lineup is determined, the battle strategy type and the game objectives involved in the battle can be determined. Since the game objectives corresponding to different battle data within the same lineup pool are essentially the same, the game objectives involved in the battle can be obtained in advance or directly from the battle data.

[0085] Specifically, the server processes each virtual object lineup in the same way. Taking one virtual object lineup as an example, the server retrieves the corresponding battle data from the lineup pool to which the virtual object lineup belongs. From the battle data, the server extracts the historical state of the virtual object lineup during the battle and obtains the game objectives completed by the virtual object lineup during the battle. Based on the matching relationship between each game objective and each historical state, the server determines the game objective corresponding to each historical state of the virtual object lineup. Based on the correspondence between the lineup pool and the battle strategy type, the server determines the battle strategy type of the virtual object lineup.

[0086] In this embodiment, virtual object lineups with the same battle strategy type and game objective are grouped together based on the lineup pool. On the one hand, it is possible to classify based on the battle strategy type, so as to query the battle strategy type corresponding to the virtual object lineup. On the other hand, it is possible to quickly determine the battle strategy type and game objective corresponding to the battle data of each virtual object lineup, thereby improving data processing efficiency.

[0087] In one embodiment, based on the matching relationship between each game objective and each historical state, the game objective corresponding to each virtual object lineup in each historical state is determined, including:

[0088] Obtain the game time period corresponding to each game objective during the battle; based on the game time period to which the corresponding time point of each historical state belongs during the battle, determine the game objective corresponding to each virtual object lineup in each historical state.

[0089] The battle process consists of multiple states arranged sequentially in time, with each historical state corresponding to a specific point in time. During the game, the game objective is neither constantly changing nor static, but rather involves completing one or more objectives within a certain timeframe. For example, the objective in the first timeframe might be to kill monsters, while the objective in the second timeframe might be to destroy defensive towers. The game objectives remain largely unchanged over a period of time, ensuring that they represent sparse data. It's worth noting that the first and second timeframes can be two consecutive time intervals or two time intervals with partial overlap, but every point in time during the battle belongs to at least one timeframe with a corresponding game objective. In other words, each historical state has at least one corresponding game objective.

[0090] Specifically, the server obtains the game time period corresponding to each game objective during the battle. For each historical state, based on the time point corresponding to the corresponding historical state during the battle and the game time period to which that time point belongs, the server determines the game objective corresponding to the virtual object lineup in that historical state, thereby obtaining the game objective corresponding to each virtual object lineup in each historical state.

[0091] In this embodiment, the server determines the relationship between the game objective and the game time period during the battle. Based on the time period to which the historical state corresponds to the time point during the battle, it can ensure that the game objective does not change within a certain period of time, ensure the sparse distribution of the game objective, and quickly and accurately determine the game objective corresponding to each historical state of the virtual object lineup, thereby improving data processing efficiency.

[0092] In one embodiment, the method for training a battle strategy model for a multiplayer game further includes: acquiring battle data from the multiplayer game, determining the lineup of virtual objects in the battle data and the game objectives achieved by the lineup of virtual objects, determining the battle strategy type corresponding to the lineup of virtual objects based on the historical state and game objectives in the battle data, dividing the battle data corresponding to the lineup of virtual objects with the same battle strategy type into the same lineup pool, and establishing a correspondence between the lineup pool and the battle strategy type.

[0093] The lineup pool is a container used to store the battle data corresponding to virtual object lineups. The server retrieves battle data from multiplayer games, obtains the virtual object lineups from the battle data through game configuration data, and determines the game objectives achieved by the virtual object lineups in that battle based on expert experience data. Generally, the battle data includes the factions of both sides, and the virtual object lineups of the two sides are different in the same battle, with each virtual object lineup achieving different game objectives.

[0094] Specifically, the battle strategy type refers to one of the two players during a battle. The battle strategy type is related to the overall state of the battle and the game objectives achieved by the targeted virtual object lineup. The server determines the battle strategy type based on the historical state of the battle data and the game objectives achieved by the virtual object lineups. Virtual object lineups with the same battle strategy type will always achieve the same game objectives, but virtual object lineups with the same game objectives may not have the same battle strategy type. By grouping battle data corresponding to virtual object lineups with the same battle strategy type into the same lineup pool, each lineup pool includes battle data corresponding to one or more virtual object lineups, and the battle strategy type and game objectives of the virtual object lineups within the same lineup pool are identical.

[0095] In this embodiment, the above processing accurately divides the battle data of virtual object lineups with the same battle strategy type and game objective into the same lineup pool. This enables the rapid and accurate determination of the battle strategy type corresponding to the virtual object lineup and the corresponding game objective in each historical state during subsequent processing, thereby improving data processing efficiency.

[0096] In one embodiment, obtaining historical states from the battle data corresponding to the virtual object lineup includes: sampling the battle data corresponding to the virtual object lineup according to the time interval conditions corresponding to the time step, to obtain historical states sorted by time step.

[0097] The time interval condition is the required time interval between any two adjacent historical states. A time step describes the historical states at different points in time. For example, if 2400 states (t=0 to t=199) need to be extracted from the start to the end of the game, and the game lasts 20 minutes, then the time interval condition for a time step can be calculated to be 0.5 seconds. Furthermore, if the time interval condition for a time step is directly set to 0.5 seconds, then the longer the game lasts, the more time steps need to be extracted.

[0098] In this embodiment, the completed battle game can be regarded as a battle video composed of multiple consecutive video frames. By extracting historical states according to the time intervals corresponding to the time steps for model training, it can be ensured that the historical states are distributed according to the same time intervals, ensuring a relatively complete time span, while effectively reducing the amount of data to be processed and improving data processing efficiency.

[0099] In one embodiment, for each combat action of each virtual object lineup obtained based on the initial combat strategy model, the response state corresponding to the corresponding combat action and the combat strategy type corresponding to the corresponding virtual object lineup are input into the game target prediction model to obtain the predicted game target corresponding to each response state of each virtual object lineup. This includes: taking each combat action of each virtual object lineup as the target combat action of the target virtual object lineup, inputting the response state corresponding to the target combat action and the combat strategy type corresponding to the target virtual object lineup into the game target prediction model to obtain candidate predicted game targets; when the candidate predicted game targets meet the prediction game target update conditions, the candidate predicted game targets are used as the predicted game targets.

[0100] Here, "target combat action" refers to the combat action currently being processed, and "target virtual object lineup" refers to the virtual object lineup corresponding to the currently processed combat action. The server's processing procedure for each combat action with each virtual object lineup is the same. The predicted game target update condition is used to determine whether the predicted game target needs to be updated. The response states input to the game target prediction model are entered sequentially according to the time step order. Therefore, the multiple candidate predicted game targets predicted by the game target prediction model are also arranged in the time step order. To ensure that the predicted targets remain unchanged over a period of time and maintain a sparse distribution, a predicted game target update condition is set, which must be met to update the predicted game target. If a candidate predicted game target does not meet the prediction game target update condition, the original predicted game target remains unchanged. If a candidate predicted game target meets the prediction game target update condition, the candidate predicted game target is adopted as the predicted game target. Specifically, the predicted game target update condition can be that the current response state reaches the corresponding game target, or that the duration of the predicted game target reaches a preset time step threshold, etc.

[0101] Specifically, taking the processing of target combat actions against a lineup of virtual objects as an example, the game system determines the changes in the virtual environment after the target combat action is executed. Based on the changed virtual environment and each virtual object in the lineup of virtual objects after executing the combat action, the system determines the response state corresponding to the target combat action. The server inputs the response state corresponding to the target combat action and the combat strategy type corresponding to the lineup of virtual objects into the game target prediction model to obtain candidate predicted game targets. When a candidate predicted game target meets the prediction game target update conditions, it is used as the predicted game target; when a candidate predicted game target does not meet the prediction game target update conditions, the original predicted game target remains unchanged.

[0102] In this embodiment, by setting a prediction game target update condition that must be met before updating the prediction game target, the candidate prediction game target is used as the prediction game target only when it meets the prediction game target update condition. This ensures that the prediction target remains unchanged for a period of time, guarantees the sparse distribution of the prediction game target, and conforms to the distribution law of the game target. Thus, based on the accurate prediction game target, the accuracy of the reward data for training the initial battle strategy model is improved.

[0103] In one embodiment, combat actions correspond to time steps in the combat process; based on the reward data corresponding to each predicted game objective, an initial combat strategy model is trained to obtain a combat strategy model, including: obtaining feedback reward data corresponding to each time step based on the intrinsic reward data corresponding to the predicted game objective at each time step and the actual reward data corresponding to the combat action at each time step; inputting the data combination consisting of the feedback reward data and response state at the same time step into the initial combat strategy model for reinforcement learning to obtain the combat strategy model.

[0104] The time step describes different historical states, with each historical state corresponding to a combat action. Therefore, combat actions correspond to time steps in the combat process. Using the time step as a standard allows for the rapid and accurate acquisition of data from various stages within the same processing flow, ensuring accurate correspondence between data.

[0105] Intrinsic reward data is the reward data corresponding to the predicted game objective, and it corresponds to the predicted game objective. In a specific application, the calculation process of intrinsic reward data includes: based on the current time step and the next time step corresponding to the predicted game objective, determining the first response state corresponding to the current time step and the second response state corresponding to the next time step; calculating the first feature distance between the first response state and the predicted game objective, and the second feature distance between the second response state and the predicted game objective; and determining the intrinsic reward data corresponding to the predicted game objective at the current time step based on the difference between the first feature distance and the second feature distance.

[0106] Actual reward data is the actual reward data provided by the game system for each combat action, based on pre-designed reward signals and the matching relationship between the combat action and the reward signals. Feedback reward data is used to represent the combined effect of actual reward data and intrinsic reward data. Specifically, feedback reward data can be the result of the superposition of actual reward data and intrinsic reward data, or the result of a weighted calculation of actual reward data and intrinsic reward data. The specific calculation method can be set according to the actual scenario requirements.

[0107] In this embodiment, feedback reward data corresponding to each time step is obtained based on intrinsic reward data and actual reward data. The data combination consisting of feedback reward data and response state at the same time step is input into the initial battle strategy model for reinforcement learning, which can improve the accuracy of the input data. Thus, a battle strategy model that makes the virtual object lineup more competitive in the competitive game can be obtained through reinforcement learning.

[0108] In one embodiment, the method for training a battle strategy model in a multiplayer game further includes: inputting the battle actions output by the initial battle strategy model for the virtual object lineup into a self-play model; and conducting a battle through the self-play model to obtain the response state corresponding to the battle actions.

[0109] Self-play is an unsupervised learning method, a reinforcement learning algorithm where machine learning learns and explores from self-play. The self-play model can predict combat actions based on the input state, control virtual objects to execute the predicted actions, and then evaluate the actions to determine the response state. In MOBA games, such as League of Legends and Honor of Kings, players are divided into two opposing teams, competing to destroy the enemy's crystal to win the game. MOBA games are characterized by complex map environments, rich characters, equipment, various hero skills, and diverse strategic gameplay.

[0110] In this embodiment, the server inputs the battle actions output by the initial battle strategy model for the virtual object lineup into the self-play model, and conducts battles through the self-play model to obtain the response state corresponding to the battle action accurately, thereby improving the effectiveness of the response state.

[0111] In one embodiment, such as Figure 4 As shown, the training methods for multiplayer game strategy models also include:

[0112] Step 402: In response to a multiplayer battle request, determine the target virtual object lineup and the corresponding battle strategy type.

[0113] Step 404: Based on the battle strategy model, determine the current battle action of each target virtual object in the target virtual object lineup.

[0114] Step 406: Input the actual response state and combat strategy type corresponding to the current combat action into the game objective prediction model to predict the current game objective.

[0115] Step 408: Feed back the current game objective and actual response status to the battle strategy model. Based on the battle strategy model, determine the next battle action of each target virtual object in the target virtual object lineup.

[0116] Multiplayer online battle games (MOBAs) are games where users on different factions control virtual objects to battle in a virtual environment, capturing or destroying enemy faction strongholds. In player-versus-computer or game-hosting scenarios, the terminal triggers a MOBA battle request. The server responds to the MOBA battle request and the virtual objects selected by the participating terminals, determining the target virtual object lineup and its corresponding battle strategy type. During the battle, the server, based on the battle strategy model, determines the current battle action of each target virtual object in the lineup. It inputs the actual response state and battle strategy type of the current battle action into the game objective prediction model to predict the current game objective. The server then feeds the current game objective and the actual response state back to the battle strategy model. Based on the battle strategy model, the server determines the next battle action for each target virtual object in the lineup. This process is repeated until a game-hosting end command is received, or the MOBA ends or is terminated.

[0117] In this embodiment, by combining the game objective prediction model with the battle strategy type, and applying it in actual human-computer battles or game hosting scenarios, the combat capability of virtual objects in the virtual object lineup can be effectively improved, thereby enhancing the user's gaming experience.

[0118] In a specific application, such as Figure 5 As shown, the training method for the battle strategy model in multiplayer games includes the following steps:

[0119] Step 502: Obtain the battle data of the multiplayer game, and determine the lineup of virtual objects in the battle data and the game objectives completed by the lineup of virtual objects.

[0120] Step 504: Based on the historical state and game objective in the battle data, determine the battle strategy type corresponding to the virtual object lineup.

[0121] Step 506: Divide the battle data corresponding to virtual object lineups with the same battle strategy type into the same lineup pool, and establish the correspondence between the lineup pool and the battle strategy type.

[0122] Step 508: For each virtual object lineup, retrieve the corresponding battle data from the lineup pool to which the virtual object lineup belongs.

[0123] Step 510: According to the time interval conditions corresponding to the time step, sample the battle data corresponding to the virtual object lineup to obtain the historical state sorted by time step, and determine the game objectives completed by the virtual object lineup during the battle.

[0124] Step 512: Obtain the game time period corresponding to each game target during the battle. Based on the game time period to which the corresponding time point of each historical state belongs during the battle, determine the game target corresponding to each historical state of the virtual object lineup.

[0125] Step 514: Determine the battle strategy type of the virtual object lineup based on the correspondence between the lineup pool and the battle strategy type.

[0126] Step 516: For each virtual object lineup, the state features corresponding to each historical state of the virtual object lineup are concatenated with the strategy type features corresponding to the battle strategy type to obtain input feature data.

[0127] Step 518: Use the game target corresponding to each input feature data as the label of the corresponding input feature data, and input the labeled input feature data corresponding to each virtual object lineup into the initial prediction model for training to obtain the game target prediction model.

[0128] Step 520: Take each combat action of each virtual object lineup as the target combat action of the target virtual object lineup, input the response state corresponding to the target combat action and the combat strategy type corresponding to the target virtual object lineup into the game target prediction model to obtain candidate predicted game targets.

[0129] Step 522: When a candidate predicted game objective meets the prediction game objective update conditions, the candidate predicted game objective is used as the predicted game objective.

[0130] Step 524: Based on the intrinsic reward data corresponding to the predicted game objective at each time step and the actual reward data corresponding to the combat action at each time step, obtain the feedback reward data corresponding to each time step.

[0131] Step 526: Input the data combination consisting of feedback reward data and response status with the same time step into the initial battle strategy model for reinforcement learning to obtain the battle strategy model.

[0132] Step 528: In response to a multiplayer game's battle request, determine the target virtual object lineup and the corresponding battle strategy type.

[0133] Step 530: Based on the battle strategy model, determine the current battle action of each target virtual object in the target virtual object lineup.

[0134] Step 532: Input the actual response state and combat strategy type corresponding to the current combat action into the game objective prediction model to predict the current game objective.

[0135] Step 534: Feed back the current game objective and actual response status to the battle strategy model. Based on the battle strategy model, determine the next battle action of each target virtual object in the target virtual object lineup.

[0136] This application also provides an application scenario in which the above-mentioned multiplayer game strategy model training method is applied. Specifically, the application of the multiplayer game strategy model training method in this scenario is as follows:

[0137] Existing tactical competitive game AIs (such as virtual objects in multiplayer online battle arena games) all employ reinforcement learning methods for training. During reinforcement learning, reward signals guide the AI's learning, essentially providing it with a set of values. Currently, most complex MOBA game AIs use human-defined dense reward signal designs. This design leads to a singular value system in the AI, resulting in limited strategic gameplay across different virtual objects. As the range of virtual object lineups expands, these singular, general strategies become inadequate for adapting to specific lineups, lacking robustness in strategic responses. The main problems with current game AI gameplay are: 1. Monotonous gameplay: Any combination of virtual objects exhibits the same strategy, failing to develop unique tactics. 2. Weak strategic adversarial ability: In training matches, opponents have similar strategies and abilities, hindering diverse strategic interactions.

[0138] In this application, as Figure 6 As shown, the entire method consists of two stages: a supervised learning process and a reinforcement learning process. The supervised learning process involves pre-training a game objective prediction model using data from different lineups and playstyles of human players. This game objective prediction model can be a meta-controller network (MC). The MC network then predicts the game objective based on the response state s returned by the game system. t Predicting future game objectives g t The distribution of game objective goals is fitted using a MC network to fit human gameplay strategies. The reinforcement learning process still uses a reinforcement learning framework to train the game AI's policy network π, introducing a hierarchical reinforcement learning framework for training on combat strategy types. The overall framework has a two-layer structure: the first layer uses a pre-trained MC network to predict future game objectives g. t And the game objective g is calculated through the reward data calculation module. t Intrinsic reward data related to completion status The second layer combines the actual reward data (reward r) involved in the original manual process. t and intrinsic reward data Training in combat strategy types.

[0139] Specifically, the server provides "prior knowledge" about specific strategies for virtual object lineups by incorporating expert data from humans. For example... Figure 7 As shown, firstly, the gameplay of the training data is divided into multiple types, each type corresponds to a battle strategy type, and each battle strategy type corresponds to a lineup pool. The server stores the battle data of virtual object lineups of the same battle strategy type into the same lineup pool. Battle data of non-special gameplay is divided into the default lineup pool.

[0140] For the same game, a unified set of game objectives (meta-goals) is defined for all battle strategies, such as killing jungle monsters, clearing minion waves, and destroying defensive towers. Different battle strategy types can correspond to certain game objectives in the set of game objectives. Specifically, the server can match the various historical states in the battle data with the game objectives in the set of game objectives. When there is a successfully matched historical state and the faction that achieved the game objective in that historical state is the specified virtual object lineup, it means that the virtual object lineup has completed the matched game objective during the battle, thus determining the game objective corresponding to the virtual object lineup.

[0141] In this context, game objectives can be states that may be reached in the future during the game, and these objectives correspond to specific game time periods. Within a given game time period, the game objective is the same for all states. However, considering the distribution of all historical states and game objectives throughout the entire battle, the game objectives are sparsely distributed.

[0142] During the training of the game objective prediction model MC, the server uses supervised learning to extract the state S at each time step from the battle data provided by the lineup pool. t And the battle strategy type F t The combined result is used as the input feature, and the game goal corresponding to that time step is used as the label. Supervised learning is performed on the MC network. Training ends when the training termination condition is met, such as reaching a threshold for the number of training iterations or a threshold for the model's output accuracy. The trained MC network can then be used to assist in reinforcement learning for combat strategy types, based on the response state S returned by the game system for a combat action at a certain time step. t And the battle strategy type F t Predict the game objective g at this time step. t .

[0143] like Figure 8 As shown, the reinforcement learning process includes: predicting the future target g using a pre-trained MC network. t And the predicted game objective g is calculated through the reward data calculation module.t Intrinsic reward data related to completion status Then, combined with the actual reward data from the original manual design, reward r t and Training of strategic online battle networks.

[0144] Among them, in order to predict the game objective g t It conforms to the rule of remaining unchanged over a period of time to ensure sparse distribution, therefore, we set the prediction of the game objective g as follows: t Updating the game objective requires meeting certain conditions. The response states are input sequentially according to the time steps; therefore, the multiple candidate game objectives predicted by the game objective prediction model are also arranged in chronological order. If a candidate game objective does not meet the update conditions, the original predicted game objective g is maintained. t If the candidate predicted game objective remains unchanged, and it meets the prediction game objective update condition, then the candidate predicted game objective is used as the predicted game objective g. t Specifically, the conditions for predicting game objective updates can be that the current response state reaches the corresponding game objective, or that the predicted duration of the game objective reaches a preset threshold of time steps.

[0145] data on intrinsic rewards The calculation process includes: first, predicting the response state S using a pre-trained MC network. t Predicting the game objective g t And through the reward data calculation module, g is calculated. t Intrinsic reward data related to completion status The calculation method for the reward data calculation module goal_reward is as follows:

[0146]

[0147] Where t represents the time step, and ||*|| represents the response state s and the predicted game objective g. t The closer the feature distance, the more likely that time step t is complete. t The faster or the easier.

[0148] The server combines the original manually designed data reward with real-world data creation. t and intrinsic reward data The training of the strategy network is guided and optimized. Specifically, the optimization can be achieved by using the PPO (Proximal Policy Optimization) reinforcement learning algorithm and training by determining the response state corresponding to each game action through self-play.

[0149] After the battle strategy model is trained, such as Figure 9 As shown, the server determines the current combat action 'a' of each target virtual object in the target virtual object lineup based on the combat strategy model. t The actual response state S corresponding to the current combat action. t Inputting the game strategy type F into the game objective prediction model MC, the current game objective g is predicted. t The current game objective g t and actual response state S t The feedback is sent to the battle strategy model. Based on the battle strategy model, the server determines the next battle action for each target virtual object in the target virtual object lineup, and executes the above process in a loop until the multiplayer battle game ends or is terminated.

[0150] Because the MC network can predict the distribution of game objective goals for different lineups, and game objective goals are key to implementing different playstyles, the distribution of game objective goals differs for different lineups. Therefore, the value system of the combat strategy module is diverse. The strategic playstyle of the trained virtual object lineup will be strongly correlated with the virtual object lineup, and the playstyle will be more diverse, resulting in better strategic combat capabilities.

[0151] It should be understood that although the steps in the flowcharts above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts above may include multiple steps or stages. These steps or stages are not necessarily completed at the same time step, but can be executed at different time steps. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0152] In one embodiment, such as Figure 10 As shown, a multiplayer game strategy model training device 1000 is provided. This device can be a software module, a hardware module, or a combination of both integrated into a computer device. Specifically, the device includes: a data acquisition module 1002, a game target prediction model training module 1004, a game target prediction module 1006, and a strategy model training module 1008, wherein:

[0153] The data acquisition module 1002 is used to acquire historical states from the battle data corresponding to each virtual object lineup, determine the game objectives corresponding to each virtual object lineup in each historical state, and acquire the battle strategy type of the virtual object lineup.

[0154] The game objective prediction model training module 1004 is used to train a game objective prediction model by taking historical states and battle strategy types as input data and game objectives as labels for the input data.

[0155] The game objective prediction module 1006 is used to input the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup into the game objective prediction model for each battle action of each virtual object lineup based on the initial battle strategy model, so as to obtain the predicted game objective corresponding to each response state of each virtual object lineup.

[0156] The battle strategy model training module 1008 is used to train the initial battle strategy model based on the reward data corresponding to each predicted game objective, so as to obtain the battle strategy model.

[0157] In one embodiment, the game target prediction model training module is further configured to, for each virtual object lineup, concatenate the state features corresponding to each historical state of the virtual object lineup with the strategy type features corresponding to the battle strategy type to obtain input feature data; use the game target corresponding to each input feature data as the label of the corresponding input feature data; and input the input feature data and label corresponding to each virtual object lineup into the initial prediction model for training to obtain the game target prediction model.

[0158] In one embodiment, the data acquisition module is further configured to, for each virtual object lineup, acquire the corresponding battle data of the virtual object lineup from the lineup pool to which the virtual object lineup belongs; extract the historical state of the virtual object lineup during the battle and the game objectives completed during the battle from the battle data; determine the game objectives corresponding to each historical state of the virtual object lineup based on the matching relationship between each game objective and each historical state; and determine the battle strategy type of the virtual object lineup based on the correspondence between the lineup pool and the battle strategy type.

[0159] In one embodiment, the data acquisition module is further configured to acquire the game time period corresponding to each game target during the battle; and based on the game time period to which the corresponding time point of each historical state belongs during the battle, determine the game target corresponding to each historical state of the virtual object lineup.

[0160] In one embodiment, the multiplayer game strategy model training device further includes a lineup pool division module, which is used to acquire multiplayer game battle data, determine the virtual object lineups in the battle data and the game objectives completed by the virtual object lineups; determine the battle strategy type corresponding to the virtual object lineups based on the historical states and game objectives in the battle data; divide the battle data corresponding to virtual object lineups with the same battle strategy type into the same lineup pool, and establish the correspondence between the lineup pool and the battle strategy type.

[0161] In one embodiment, the data acquisition module is further configured to sample the battle data corresponding to the virtual object lineup according to the time interval conditions corresponding to the time step, so as to obtain the historical state sorted by time step.

[0162] In one embodiment, the game target prediction module is further configured to take each combat action of each virtual object lineup as the target combat action of the target virtual object lineup, input the response state corresponding to the target combat action and the combat strategy type corresponding to the target virtual object lineup into the game target prediction model to obtain candidate predicted game targets; when the candidate predicted game targets meet the prediction game target update conditions, the candidate predicted game targets are used as the predicted game targets.

[0163] In one embodiment, combat actions correspond to time steps in the combat process; the combat strategy model training module is also used to obtain feedback reward data corresponding to each time step based on the intrinsic reward data corresponding to the predicted game objective at each time step and the actual reward data corresponding to the combat action at each time step; the data combination consisting of the feedback reward data and response state at the same time step is input into the initial combat strategy model for reinforcement learning to obtain the combat strategy model.

[0164] In one embodiment, the method for training a battle strategy model for a multiplayer game further includes an intrinsic reward data calculation module, used to determine a first response state corresponding to the current time step and a second response state corresponding to the next time step based on the current time step and the next time step corresponding to the predicted game objective; calculate a first feature distance between the first response state and the predicted game objective and a second feature distance between the second response state and the predicted game objective; and determine the intrinsic reward data corresponding to the predicted game objective at the current time step based on the difference between the first feature distance and the second feature distance.

[0165] In one embodiment, the method for training a battle strategy model in a multiplayer game further includes a response state determination module, which is used to input the battle actions output by the initial battle strategy model for the virtual object lineup into the self-play model; and to obtain the response state corresponding to the battle actions by playing a battle through the self-play model.

[0166] In one embodiment, the method for training a battle strategy model for a multiplayer game further includes a model application module, used to respond to a battle request from a multiplayer game, determine the lineup of target virtual objects and the battle strategy type corresponding to the lineup of target virtual objects; based on the battle strategy model, determine the current battle action of each target virtual object in the lineup of target virtual objects; input the actual response state and battle strategy type corresponding to the current battle action into a game target prediction model to predict the current game target; feed back the current game target and the actual response state to the battle strategy model, and based on the battle strategy model, determine the next battle action of each target virtual object in the lineup of target virtual objects.

[0167] For specific embodiments of the multiplayer game strategy model training device, please refer to the embodiments of the multiplayer game strategy model training method described above, which will not be repeated here. Each module in the aforementioned multiplayer game strategy model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding actions of each module.

[0168] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores training data for multiplayer game strategy models. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for training a multiplayer game strategy model.

[0169] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0170] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0171] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0172] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0174] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0175] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training a battle strategy model in a multiplayer battle game, characterized in that, The method includes: For each virtual object lineup, historical states are obtained from the battle data corresponding to the virtual object lineup, the game objective corresponding to each of the historical states is determined, and the battle strategy type of the virtual object lineup is obtained; the battle strategy type is used to characterize the type of strategy and tactics adopted in the entire game battle process. Using the historical state and the battle strategy type as input data, and the game objective as the label of the input data, a game objective prediction model is trained. For each battle action of each virtual object lineup obtained based on the initial battle strategy model, the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup. Based on the reward data corresponding to each of the predicted game objectives, the initial battle strategy model is trained to obtain the battle strategy model.

2. The method according to claim 1, characterized in that, The process of training a game objective prediction model using the historical state and the battle strategy type as input data, and the game objective as the label of the input data, includes: For each virtual object lineup, the state features corresponding to each historical state of the virtual object lineup are concatenated with the strategy type features corresponding to the battle strategy type to obtain input feature data. The game objective corresponding to each of the input feature data is used as the label of the corresponding input feature data; The input feature data and labels corresponding to each virtual object lineup are input into the initial prediction model for training, thus obtaining the game target prediction model.

3. The method according to claim 1, characterized in that, For each virtual object lineup, the historical states are obtained from the corresponding battle data of the virtual object lineup, the game objective corresponding to each of the historical states of the virtual object lineup is determined, and the battle strategy type of the virtual object lineup is obtained, including: For each virtual object lineup, retrieve the corresponding battle data from the lineup pool to which the virtual object lineup belongs; From the battle data, extract the historical state of the virtual object lineup during the battle and the game objectives completed during the battle; Based on the matching relationship between each game objective and each historical state, the game objective corresponding to each of the virtual object lineups in each of the historical states is determined. Based on the correspondence between the lineup pool and the battle strategy type, the battle strategy type of the virtual object lineup is determined.

4. The method according to claim 3, characterized in that, The process of determining the game objectives corresponding to the virtual object lineup in each historical state based on the matching relationship between each game objective and each historical state includes: Obtain the corresponding game time segment for each game objective during the battle; Based on the game time period corresponding to each historical state during the battle, the game objectives corresponding to each of the virtual object lineups in each historical state are determined.

5. The method according to claim 3, characterized in that, The method further includes: Acquire battle data from a multiplayer game, and determine the lineup of virtual objects in the battle data and the game objectives achieved by the lineup of virtual objects. Based on the historical state in the battle data and the game objective, determine the battle strategy type corresponding to the virtual object lineup; The battle data corresponding to virtual object lineups with the same battle strategy type are divided into the same lineup pool, and a correspondence is established between the lineup pool and the battle strategy type.

6. The method according to claim 1, characterized in that, The step of obtaining historical states from the battle data corresponding to the virtual object lineup includes: Based on the time interval conditions corresponding to the time step, the battle data corresponding to the virtual object lineup is sampled to obtain the historical state sorted by time step.

7. The method according to claim 1, characterized in that, For each combat action of each virtual object lineup obtained based on the initial combat strategy model, the response state corresponding to the corresponding combat action and the combat strategy type corresponding to the corresponding virtual object lineup are input into the game objective prediction model to obtain the predicted game objective corresponding to each response state of each virtual object lineup, including: Each combat action of each virtual object lineup is taken as the target combat action of the target virtual object lineup. The response state corresponding to the target combat action and the combat strategy type corresponding to the target virtual object lineup are input into the game target prediction model to obtain candidate predicted game targets. When the candidate predicted game objective meets the prediction game objective update conditions, the candidate predicted game objective is used as the predicted game objective.

8. The method according to claim 1, characterized in that, The combat actions correspond to the time steps in the combat process; The process of training the initial battle strategy model based on the reward data corresponding to each predicted game objective to obtain the battle strategy model includes: Based on the intrinsic reward data corresponding to the predicted game objective at each time step and the actual reward data corresponding to the combat action at each time step, the feedback reward data corresponding to each time step is obtained. The data combination consisting of feedback reward data and response status at the same time step is input into the initial battle strategy model for reinforcement learning to obtain the battle strategy model.

9. The method according to claim 8, characterized in that, The method further includes: Based on the current time step and the next time step corresponding to the predicted game objective, determine the first response state corresponding to the current time step and the second response state corresponding to the next time step; Calculate the first feature distance between the first response state and the predicted game target, and the second feature distance between the second response state and the predicted game target, respectively; Based on the difference between the first feature distance and the second feature distance, the intrinsic reward data corresponding to the predicted game objective at the current time step is determined.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: The initial battle strategy model outputs battle actions for the virtual object lineup, which are then input into the game model. By engaging in a game through a self-play model, the response state corresponding to the game action is obtained.

11. The method according to any one of claims 1 to 9, characterized in that, The method further includes: In response to a battle request in a multiplayer game, determine the target virtual object lineup and the battle strategy type corresponding to the target virtual object lineup; Based on the aforementioned battle strategy model, determine the current battle action of each target virtual object in the target virtual object lineup; The actual response state corresponding to the current combat action and the combat strategy type are input into the game objective prediction model to predict the current game objective. The current game objective and the actual response status are fed back to the battle strategy model. Based on the battle strategy model, the next battle action of each target virtual object in the target virtual object lineup is determined.

12. A training device for a battle strategy model in a multiplayer battle game, characterized in that, The device includes: The data acquisition module is used to acquire historical states from the battle data corresponding to each virtual object lineup, determine the game objective corresponding to each virtual object lineup in each historical state, and acquire the battle strategy type of the virtual object lineup; the battle strategy type is used to characterize the type of strategy and tactics adopted in the entire game battle process. The game objective prediction model training module is used to train a game objective prediction model by taking the historical state and the battle strategy type as input data and the game objective as the label of the input data. The game objective prediction module is used to input the response state corresponding to the corresponding battle action and the battle strategy type corresponding to the corresponding virtual object lineup into the game objective prediction model for each battle action of each virtual object lineup obtained based on the initial battle strategy model, so as to obtain the predicted game objective corresponding to each response state of each virtual object lineup. The battle strategy model training module is used to train the initial battle strategy model based on the reward data corresponding to each of the predicted game objectives, so as to obtain the battle strategy model.

13. The multiplayer game strategy model training device according to claim 12, characterized in that, The game target prediction model training module is further used to, for each virtual object lineup, concatenate the state features corresponding to each historical state of the virtual object lineup with the strategy type features corresponding to the battle strategy type to obtain input feature data; use the game target corresponding to each input feature data as the label of the corresponding input feature data; and input the input feature data and label corresponding to each virtual object lineup into the initial prediction model for training to obtain the game target prediction model.

14. The multiplayer game strategy model training device according to claim 12, characterized in that, The data acquisition module is further configured to, for each virtual object lineup, acquire the corresponding battle data of the virtual object lineup from the lineup pool to which the virtual object lineup belongs; extract the historical state of the virtual object lineup during the battle and the game objectives completed during the battle from the battle data; determine the game objectives corresponding to each historical state of the virtual object lineup based on the matching relationship between each game objective and each historical state; and determine the battle strategy type of the virtual object lineup based on the correspondence between the lineup pool and the battle strategy type.

15. The multiplayer game strategy model training device according to claim 14, characterized in that, The data acquisition module is also used to acquire the game time period corresponding to each game target during the battle; based on the game time period to which the time point corresponding to each historical state belongs during the battle, the module determines the game target corresponding to each of the virtual object lineups in each of the historical states.

16. The multiplayer game strategy model training device according to claim 14, characterized in that, The device further includes a lineup pool division module, used to acquire battle data of multiplayer games, determine the virtual object lineups in the battle data and the game objectives completed by the virtual object lineups; determine the battle strategy type corresponding to the virtual object lineups based on the historical states in the battle data and the game objectives; divide the battle data corresponding to virtual object lineups with the same battle strategy type into the same lineup pool, and establish a correspondence between the lineup pools and the battle strategy types.

17. The multiplayer game strategy model training device according to claim 12, characterized in that, The data acquisition module is also used to sample the battle data corresponding to the virtual object lineup according to the time interval conditions corresponding to the time step, so as to obtain the historical state sorted by time step.

18. The multiplayer game strategy model training device according to claim 12, characterized in that, The game target prediction module is further configured to take each combat action of each virtual object lineup as the target combat action of the target virtual object lineup, input the response state corresponding to the target combat action and the combat strategy type corresponding to the target virtual object lineup into the game target prediction model to obtain candidate predicted game targets; when the candidate predicted game targets meet the prediction game target update conditions, the candidate predicted game targets are used as the predicted game targets.

19. The multiplayer game strategy model training device according to claim 12, characterized in that, The combat actions correspond to time steps in the combat process; the combat strategy model training module is also used to obtain feedback reward data corresponding to each time step based on the intrinsic reward data corresponding to the predicted game objective at each time step and the actual reward data corresponding to the combat action at each time step; the data combination consisting of the feedback reward data and response state at the same time step is input into the initial combat strategy model for reinforcement learning to obtain the combat strategy model.

20. The multiplayer game strategy model training device according to claim 19, characterized in that, The device further includes an intrinsic reward data calculation module, used to determine a first response state corresponding to the current time step and a second response state corresponding to the next time step based on the current time step and the next time step corresponding to the predicted game objective; calculate a first feature distance between the first response state and the predicted game objective and a second feature distance between the second response state and the predicted game objective; and determine the intrinsic reward data corresponding to the predicted game objective at the current time step based on the difference between the first feature distance and the second feature distance.

21. The multiplayer game strategy model training device according to claim 12, characterized in that, The device further includes a response state determination module, which is used to input the battle action output by the initial battle strategy model for the virtual object lineup into the self-play model; and to obtain the response state corresponding to the battle action by playing a battle through the self-play model.

22. The multiplayer game strategy model training device according to claim 12, characterized in that, The device also includes a model application module, used to respond to a battle request in a multiplayer game, determine the target virtual object lineup and the battle strategy type corresponding to the target virtual object lineup; and determine the current battle action of each target virtual object in the target virtual object lineup based on the battle strategy model. The actual response state corresponding to the current combat action and the combat strategy type are input into the game objective prediction model to predict the current game objective. The current game objective and the actual response status are fed back to the battle strategy model. Based on the battle strategy model, the next battle action of each target virtual object in the target virtual object lineup is determined.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

24. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images