Strategy model generation method, game copy execution method and device

Abstract

The embodiment of the invention provides a strategy model generation method and a game copy execution method and device.The strategy model generation method comprises the steps that a sample data set corresponding to a target copy is obtained, and the sample data set comprises a sample state set, a sample behavior set and a sample excitation set; constructing an initial behavior model based on the sample state set, the sample behavior set and the sample excitation set; based on each piece of sample state data in the sample state set, predicting a corresponding first sample behavior through the initial behavior model, executing the first sample behavior in the target copy, and obtaining corresponding first updated sample state data and first sample excitation data, updating the initial behavior model by using the first update sample state data and the first sample excitation data; and obtaining the replica strategy model under the condition that the update stop condition is met. The strategy model with high universality is generated through dynamic training, the requirements of different players can be met, and the game experience and the automation level are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a strategy model generation method, a game copy execution method and a device. Background Art

[0002] As an important part of the game, game copies require players to invest a lot of time and energy. However, in the fast-paced modern life, players can often only use fragmented time to play games, resulting in slow game progress and reduced experience.

[0003] In the related art, game planners have designed a variety of automatic execution copy hang-up methods, but most of them are designed based on the planner's experience and lack consideration of individual differences among players. Different players have differences in levels, genres, and equipment. The same automatic execution strategy is difficult to meet personalized needs and sometimes even reduces the gaming experience.

[0004] Therefore, a general strategy model generation method is urgently needed. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a strategy model generation method. One or more embodiments of the present invention also relate to a game copy execution method, a strategy model generation device, a game copy execution device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of the present invention, a policy model generation method is provided, comprising: Obtain a sample data set corresponding to the target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; Constructing an initial behavior model based on a sample state set, a sample behavior set, and a sample incentive set; Based on each sample state data in the sample state set, predict the corresponding first sample behavior through the initial behavior model, execute the first sample behavior in the target copy, obtain the corresponding first updated sample state data and first sample incentive data, and update the initial behavior model using the first updated sample state data and the first sample incentive data; When the update stop condition is met, the replica policy model is obtained.

[0007] According to a second aspect of an embodiment of the present invention, a game copy execution method is provided, comprising: In response to a copy automatic execution instruction issued by a front-end user, determining a copy identifier of a target copy and a user identifier of the front-end user; Obtaining a replica policy model corresponding to the target replica based on the replica identifier, wherein the replica policy model is obtained based on a policy model generation method; Obtain the state information of the target replica based on the user identifier, and input the state information into the replica strategy model to predict the action instruction; Executes the target copy based on the action instruction.

[0008] According to a third aspect of an embodiment of the present invention, there is provided a policy model generating device, comprising: A first acquisition module is configured to acquire a sample data set corresponding to a target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; A construction module, configured to construct an initial behavior model based on a sample state set, a sample behavior set, and a sample stimulus set; An updating module is configured to predict a corresponding first sample behavior through an initial behavior model based on each sample state data in the sample state set, execute the first sample behavior in a target replica, obtain corresponding first updated sample state data and first sample incentive data, and update the initial behavior model using the first updated sample state data and the first sample incentive data; The stop module is configured to obtain the replica strategy model when the update stop condition is met.

[0009] According to a fourth aspect of an embodiment of the present invention, a game copy execution device is provided, comprising: A determination module, configured to determine the copy identifier of the target copy and the user identifier of the front-end user in response to the copy automatic execution instruction issued by the front-end user; A second acquisition module is configured to acquire a replica policy model corresponding to the target replica based on the replica identifier, wherein the replica policy model is obtained based on a policy model generation method; An input module is configured to obtain state information of a target replica based on a user identifier, and input the state information into a replica strategy model to predict an action instruction; The execution module is configured to execute the target copy based on the action instruction.

[0010] According to a fifth aspect of an embodiment of the present invention, there is provided a computing device, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned strategy model generation method and game copy execution method are implemented.

[0011] According to a sixth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned strategy model generation method and game copy execution method.

[0012] According to a seventh aspect of an embodiment of the present invention, there is provided a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned strategy model generation method and game copy execution method.

[0013] An embodiment of the present invention realizes obtaining a sample data set corresponding to a target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; constructing an initial behavior model based on the sample state set, the sample behavior set, and the sample incentive set; predicting the corresponding first sample behavior through the initial behavior model based on each sample state data in the sample state set, executing the first sample behavior in the target replica, obtaining the corresponding first updated sample state data and the first sample incentive data, and updating the initial behavior model using the first updated sample state data and the first sample incentive data; obtaining a replica strategy model when the update stop condition is met. An initial behavior model is constructed by collecting and analyzing sample data of the target replica, including the sample state set, the sample behavior set, and the sample incentive set. Based on this model, the system can predict and execute the behavior that best matches the current state, and update the model according to the execution result. This process is iterated repeatedly until the model reaches the optimal state, and finally generates a personalized replica strategy model. Taking into account the individual differences of players, the strategy can also be dynamically adjusted to ensure that each player can obtain the best gaming experience, which can significantly improve the automation level of the game and the player satisfaction, so that players can efficiently advance the game progress in fragmented time and enjoy a smooth gaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural interaction diagram of a game copy execution system provided by an embodiment of the present invention; Figure 2 is a flow chart of a strategy model generation method provided by an embodiment of the present invention; Figure 3 is a flow chart of a game copy execution method provided by an embodiment of the present invention; Figure 4 is a process flow chart of a strategy model generation method provided by an embodiment of the present invention; Figure 5 It is a structural schematic diagram of a strategy model generating device provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of a game copy execution device provided by an embodiment of the present invention; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0016] The terms used in one or more embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present invention. The singular forms of "a", "said" and "the" used in one or more embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present invention, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] First, the terms involved in one or more embodiments of the present invention are explained.

[0020] Q-Learning: is a model-free reinforcement learning algorithm that aims to learn the optimal strategy through interaction with the environment. In Q-Learning, the agent learns how to choose actions in different states to maximize the cumulative reward through trial and error. Q-Learning does not require prior knowledge of the dynamic characteristics of the environment, but gradually approaches the optimal strategy by constantly trying and updating the Q value.

[0021] State: represents the situation or position of the agent in the environment. The state can be any observable feature of the environment, such as position, health, equipment status, etc. in the game. The state is the basis for the agent to make decisions, and different states may lead to different optimal actions.

[0022] Action: is the behavior that the agent can take in a certain state. The choice of action affects the state of the environment and may bring rewards or penalties. In different states, the agent can choose different actions. For example, in a game, an action can be moving, attacking, using a skill, etc.

[0023] Reward: It is the immediate feedback that the agent gets from the environment after taking an action. Rewards can be positive (indicating a good result) or negative (indicating a bad result). Rewards are used to evaluate the quality of actions and guide the agent to learn how to choose the best action. For example, in a game, defeating an enemy can get a positive reward, while being hurt may get a negative reward.

[0024] Q-table: A two-dimensional table that stores the expected reward value (Q value) for each action in each state. The rows of the Q-table represent states, the columns represent actions, and the value in each cell represents the expected cumulative reward for taking that action in that state. Q-Learning gradually learns the optimal strategy by continuously updating the values ​​in the Q-table.

[0025] The gaming industry has experienced rapid development, and various styles of games have emerged one after another, attracting more and more players. As a very common way to play in games, each time a copy is entered, the player will enter an independent map space and obtain rich rewards by completing specific tasks and defeating the copy's boss. As the game operation time increases, the number of game copies continues to increase, and players need to invest more time and energy to complete these contents. However, in the fast-paced life of modern society, it is difficult for players to take out large blocks of time to play games continuously. Fragmented game time leads to slow game progress and affects the gaming experience.

[0026] On this basis, the copy AI can help players hang up and fight monsters and complete copy tasks. However, the copy AI generation methods currently on the market mainly rely on the experience and ideas of planners, and there are many problems: First, the copy AI process is not reusable, and each time a new copy appears, the process needs to be redeveloped, which increases development costs and time. Secondly, the development process is complicated, and procedural programming makes code maintenance difficult, difficult to expand and maintain. Once there is a small change, a large amount of code often needs to be refactored. Finally, the copy AI made based on planning experience cannot adapt to all players. The specific situation of each player in the game (such as level, genre, equipment, etc.) is quite different, so different strategies should be adopted in the same scenario. Existing AI methods are difficult to meet this personalized demand.

[0027] In view of the above problems, the present invention provides a strategy model generation method. One or more embodiments of the present invention also relate to a game copy execution system, a game copy execution method, a strategy model generation device, a game copy execution device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0028] See also Figure 1 , Figure 1 The structure interaction diagram of a game copy execution system provided by an embodiment of the present invention is shown in FIG. The game copy execution system includes a data end, a server end and a user end, wherein the data end is used to provide a sample data set to the server end; the server end is used to obtain a sample data set corresponding to a target copy at the data end, wherein the sample data set includes a sample state set, a sample behavior set and a sample incentive set, an initial behavior model is constructed based on the sample state set, the sample behavior set and the sample incentive set, and the corresponding first sample behavior is predicted by the initial behavior model based on each sample state data in the sample state set, the first sample behavior is executed in the target copy, the corresponding first updated sample state data and the first sample incentive data are obtained, and the initial behavior model is updated using the first updated sample state data and the first sample incentive data, and the copy strategy model is obtained when the update stop condition is met; the user end is used to respond to the copy automatic execution instruction issued by the front-end user, determine the copy identifier of the target copy and the user identifier of the front-end user, obtain the copy strategy model corresponding to the target copy based on the copy identifier, wherein the copy strategy model is obtained based on the strategy model generation method, the state information of the target copy is obtained based on the user identifier, and the state information is input into the copy strategy model to predict the action instruction, and the target copy is executed based on the action instruction.

[0029] Applied to the game copy execution system, through the collaborative work of the data end, the server end and the user end, an efficient copy automatic execution function is realized. The data end provides a rich sample data set, and the server end trains a highly versatile copy strategy model based on these data sets, which can adapt to the needs of different players. The user end calls the corresponding copy strategy model according to the automatic execution instructions issued by the front-end user, predicts and executes the optimal action instructions, thereby efficiently advancing the game progress in fragmented time and improving the player's gaming experience and satisfaction.

[0030] See also Figure 2 , Figure 2 It is a flow chart of a strategy model generation method provided by an embodiment of the present invention, which specifically includes the following steps.

[0031] Step 202: Obtain a sample data set corresponding to the target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set.

[0032] The sample data set refers to a data set containing a series of player behaviors and their results in the target instance. The sample state set records various status information of the player in the instance, such as location, health, equipment status, etc. The sample behavior set records the behaviors taken by the player in a specific state, such as moving, attacking, using skills, etc. The sample incentive set records the feedback obtained by the player after each behavior, including positive rewards (such as experience points, props) and negative penalties (such as reduced health).

[0033] In practical applications, the target copy must first be determined, that is, the specific copy from which data is to be collected. The system monitors the player's activities in the copy and automatically records every state change, every action, and the results of each action. For example, when a player enters a copy, the system begins to record the player's starting position, health value, and other information, which constitutes part of the sample state set. Subsequently, whenever the player performs an action, such as moving forward or attacking an enemy, the system records the action and the conditions under which it occurs, forming a sample behavior set. Next, the system also records the player's reward or punishment for the action, such as experience points gained after killing an enemy or health points lost after being attacked. This information constitutes a sample incentive set. Throughout the process, the system continuously collects data until the player completes the copy challenge or exits the copy. For the step of obtaining a sample data set, one implementation method is to directly collect data from online players through the game server, and another implementation method is to use historical data, that is, to extract past player behavior records in the same copy from the database.

[0034] In a specific embodiment of the present invention, it is assumed that the target copy is "Dark Forest". After the system is started, it starts to monitor all players who enter the "Dark Forest". Whenever a player enters the copy, the system begins to record the player's initial state, including the player's position, health points, equipment and other information. Subsequently, the system continues to track every action of the player in the copy, such as the player chooses to move 10 meters north, use sword skills to attack the monster in front, etc. These actions are recorded together with the environmental information when they occur. At the same time, the system also records the results of each player's action, such as the experience points gained after attacking the monster, the health points lost after being counterattacked by the monster, etc. The system continues to record until the player completes the copy or exits the copy. The collected data will be organized into a structured sample data set for subsequent analysis and model training.

[0035] Furthermore, a sample data set corresponding to the target copy is obtained, including: based on the copy identifier of the target copy, a completed copy data set is obtained, wherein the completed copy data set is the copy data of the target copy that has passed the level; the completed copy data set is parsed to obtain behavior data representing user behavior as a sample behavior set, state data representing the target copy during the level passing process as a sample state set, and feedback information corresponding to the behavior data as a sample incentive set; the sample state set, the sample behavior set, and the sample incentive set are used as the sample data set corresponding to the target copy.

[0036] The sample data set refers to a data set containing a series of player behaviors and their results in the target instance. The sample state set records various status information of the player in the instance, such as location, health, equipment status, etc. The sample behavior set records the behaviors taken by the player in a specific state, such as moving, attacking, using skills, etc. The sample incentive set records the feedback information obtained by the player after each behavior, including positive rewards (such as experience points, props) and negative penalties (such as reduced health).

[0037] In actual applications, the system first obtains the completed copy data set from the database based on the copy ID of the target copy. The completed copy data set contains the data of all players who have successfully cleared the target copy. Specifically, the system will query the database based on the copy ID (such as "1001" corresponds to the "Dark Forest" copy) and extract all player records who have successfully cleared the copy. These records usually include every step of the player's operation, state changes, and feedback information.

[0038] Next, the system parses the replica dataset and extracts three main parts: Sample behavior set: The system extracts all behaviors performed by the player during the level-clearing process from the completed instance data set, such as movement, attack, and skill use. These behavior data represent the specific operations of the player.

[0039] Sample state set: The system extracts various state information of the player during the clearance process from the completed instance data set, such as position, health value, equipment status, etc. These state data represent the specific situation of the player at each time node.

[0040] Sample incentive set: The system extracts the feedback information obtained by the player after each action is performed from the completed copy data set, such as experience value increase, health value reduction, etc. This feedback information represents the effect of the player's behavior.

[0041] Finally, the system integrates the extracted sample state set, sample behavior set, and sample incentive set into the sample data set corresponding to the target instance. These data sets will be used for subsequent model training and optimization to help the system better understand and predict the player's behavior in the instance.

[0042] For the step of obtaining the completed copy data set, one implementation method is to extract it from the real-time data of the game server to ensure the real-time and accuracy of the data. Another implementation method is to extract it from historical data, which is suitable for offline analysis and model training. In either case, the system needs to ensure the integrity and consistency of the data in order to accurately parse and extract useful information.

[0043] In a specific embodiment of the present invention, assume that the target copy is "Dark Forest" and the copy identifier is "1001". The system first obtains the data of all players who have successfully passed the "Dark Forest" from the database to form a completed copy data set. Assume that there are 1000 records of successful clearance in the database, and each record includes each step of the player's operation, state change and feedback information.

[0044] The system parses these records and extracts the sample behavior set, sample state set, and sample incentive set: Sample behavior set: The system extracts all behaviors performed by players during the level-clearing process, such as "move forward 10 meters", "use fireball to attack monsters", etc.

[0045] Sample status set: The system extracts the status information of all players during the clearance process, such as "at the dungeon entrance", "health value 100 / 100", "equipment level medium", etc.

[0046] Sample incentive set: The system extracts the feedback information obtained by all players after each action is performed, such as "gaining 10 experience points", "health points reduced by 5 points", etc.

[0047] The system integrates these extracted data into a sample data set corresponding to the target replica "Dark Forest". For example, a record in the sample data set can be as follows: Sample status: [Dungeon entrance, HP 100 / 100, medium equipment level] Sample behavior: [Move forward 10 meters] Sample incentive: [Gain 10 experience points] Based on this, by obtaining and parsing the completed copy data set of the target copy, the system can build a complete sample data set, providing rich data support for subsequent model training and optimization. This method can not only help the system better understand and predict the behavior of players in the copy, but also improve the intelligence level of copy automation and enhance the player's gaming experience. In addition, by extracting useful information from a large amount of successful clearance data, the system can ensure that the trained model has high accuracy and reliability, and can perform well in practical applications.

[0048] Step 204: construct an initial behavior model based on the sample state set, the sample behavior set and the sample stimulus set.

[0049] The initial behavior model is the Q-table, which is used to store the expected reward value that can be obtained by performing an action in a given state. Each item in the Q-table corresponds to a state-action pair, and its value reflects the estimated long-term benefit brought by the action in a specific state.

[0050] In practical applications, the process of building an initial behavior model begins with the collected sample data sets, including sample state sets, sample behavior sets, and sample incentive sets. First, the system needs to define the state space and action space. The state space covers all states that the player may encounter in the game, such as the player's position, health, equipment status, etc. The action space contains all actions that the player can perform, such as moving, attacking, using skills, etc. Next, based on these definitions, the system creates an empty Q table, where rows represent states, columns represent actions, and initial values ​​are usually set to 0 or a small positive number to encourage exploration of unknown state-action pairs.

[0051] The system then uses each record in the sample data set (i.e., a state, an action, and an incentive) to update the Q-table. For each record, the system finds the corresponding state-action pair and adjusts the Q value of the pair based on the reward value in the sample incentive set. The adjustment is usually made using the update formula in the Q-learning algorithm, where the new Q value equals the old Q value plus the learning rate multiplied by (the immediate reward plus the discount factor multiplied by the maximum Q value in the next state minus the old Q value). In this way, the Q-table is gradually filled to reflect the expected benefit of performing a specific action in a specific state.

[0052] In a specific embodiment of the present invention, it is assumed that the target copy is a "Dragon Cave". The system first defines the state space, including the player's position, health, equipment level, etc., and the action space, including the direction of movement, attack type, skill use, etc. Then, the system creates an empty Q table, and the initial Q value of each state-action pair is set to 0. The system begins to process each record in the sample data set. For example, when the player is at the entrance of the dragon cave, the health is full, and the equipment level is medium, the action of moving forward 10 meters is selected, and then a small monster is encountered and 10 experience points are obtained as a reward. The system updates the Q value of the corresponding state-action pair in the Q table according to this record, and the calculation formula is: new Q value = old Q value + learning rate * (10 + discount factor * maximum Q value in the next state - old Q value). The system repeats this process, processes all records in the sample data set, and gradually improves the Q table.

[0053] In order to further optimize the update process, the system adopts a greedy strategy. Specifically, the system calculates the value probability of each action and counts the probability of each action occurring when players in other environments pass through the copy (such as external network players). This is done so that more optimal actions can be selected each time an update is performed, thereby speeding up the update progress. In this way, the system can converge to the optimal strategy faster and improve learning efficiency. Specifically, when updating the Q table, the system not only relies on a fixed update formula, but also gives priority to updating those actions with higher values ​​based on the distribution of the current Q value, thereby accelerating the optimization process of the Q table. This greedy strategy helps the system find a better balance between exploration and exploitation, ensuring better performance within a limited training time.

[0054] Furthermore, an initial behavior model is constructed based on the sample state data set, the sample behavior data set and the sample incentive data set, including: based on the sample incentive set, determining the behavior value corresponding to each sample behavior data in the sample behavior set; based on the sample incentive set, determining the state value corresponding to each sample state data in the sample state set, wherein the state value is used to describe the completion progress of the target copy; and constructing an initial behavior model based on the sample state set, the sample behavior set, the behavior value and the state value.

[0055] Among them, the sample state data set records various status information of the player in the copy, such as position, health, equipment status, etc. The sample behavior data set records the behavior taken by the player in a specific state, such as moving, attacking, using skills, etc. The sample incentive data set records the feedback information obtained by the player after each behavior, including positive rewards (such as experience points, props) and negative penalties (such as reduced health). The initial behavior model refers to a model used to store the expected reward value that can be obtained by performing an action in a given state, usually expressed as a Q table.

[0056] In practical applications, the system first determines the behavior value corresponding to each sample behavior data in the sample behavior set based on the sample incentive set. The behavior value reflects the immediate reward or penalty obtained after performing a certain behavior. For example, the system can assign an initial value to each behavior, such as 0 for normal movement, 5 for attacking with skills, and -10 for being attacked. These initial values ​​can be determined based on experience or through statistical analysis.

[0057] Next, the system determines the state value corresponding to each sample state data in the sample state set based on the sample incentive set. The state value is used to describe the completion progress of the target copy and reflects the overall performance of the player in a specific state. For example, the system can assign an initial value to each state, such as a state close to the end of the copy has a higher value, while a state just entering the copy has a lower value. These initial values ​​can also be determined through statistical analysis, for example, the closer the state is to the end of the copy, the higher its value.

[0058] Finally, the system builds an initial behavior model based on the sample state set, sample behavior set, behavior value, and state value. Specifically, the system creates a Q table where rows represent states, columns represent behaviors, and the value of each cell represents the expected reward value for performing a specific behavior in a specific state. The value of the initial Q table can be set to a combination of behavior value and state value, for example, Q value = behavior value + state value. In this way, the system initializes the Q table to provide a basis for subsequent training and optimization.

[0059] For the steps to determine the behavior value and state value, one implementation method is to calculate the average reward or penalty for each behavior and state by statistically analyzing the existing sample incentive data. Another implementation method is to manually assign an initial value to each behavior and state based on expert experience or preset rules. In either case, the system needs to ensure the rationality and consistency of the initial value in order to accurately reflect the player's behavior and state in the copy.

[0060] In a specific embodiment of the present invention, assuming that the target copy is "Dark Forest", the system obtains 1,000 records of successful clearance from the database, forming a sample state data set, a sample behavior data set, and a sample incentive data set. The system first determines the behavior value corresponding to each sample behavior data in the sample behavior set based on the sample incentive set. For example, the system statistically finds that the average reward for using fireball to attack monsters is 5 experience points, while the average penalty for being attacked by monsters is -10 health points. Therefore, the system assigns a value of 5 to the behavior of using fireball to attack monsters, and a value of -10 to the behavior of being attacked by monsters.

[0061] Next, the system determines the state value corresponding to each sample state data in the sample state set based on the sample incentive set. For example, the system statistically finds that the average reward for the state close to the end of the copy is higher, while the average reward for the state just entering the copy is lower. Therefore, the system assigns a higher state value, such as 100, to the state close to the end of the copy, and a lower state value, such as 10, to the state just entering the copy.

[0062] Finally, the system builds an initial behavior model based on the sample state set, sample behavior set, behavior value and state value. The system creates a Q table, where rows represent states, columns represent behaviors, and the value of each cell represents the expected reward value for performing a specific behavior in a specific state. For example, for the state "approaching the end of the copy" and the behavior "using fireball to attack monsters", the Q value = behavior value 5 + state value 100 = 105. The system initializes the Q table in this way, providing a basis for subsequent training and optimization.

[0063] Based on this, by constructing an initial behavior model based on a sample state data set, a sample behavior data set, and a sample incentive data set, the system can assign a reasonable initial Q value to each state-behavior pair. This method not only provides a solid foundation for subsequent training and optimization, but also helps the system better understand and predict the behavior of players in the dungeon, improve the intelligence level of dungeon automation, and enhance the player's gaming experience. In addition, by combining behavior value and state value, the system can more comprehensively evaluate the expected reward of each state-behavior pair, making the model more accurate and reliable in practical applications.

[0064] Step 206: Based on each sample state data in the sample state set, predict the corresponding first sample behavior through the initial behavior model, execute the first sample behavior in the target copy, obtain the corresponding first updated sample state data and first sample incentive data, and use the first updated sample state data and the first sample incentive data to update the initial behavior model.

[0065] The sample state data refers to the state information of the player at a certain moment in the copy, such as position, health, equipment, etc. The initial behavior model is the Q table, which is used to store the expected reward value that can be obtained by performing a certain action in a given state. The first sample behavior refers to the behavior that the player should take in a specific state predicted by the Q table. The first updated sample state data is the new state information of the player after executing the first sample behavior. The first sample incentive data refers to the reward or punishment obtained by the player after executing the first sample behavior.

[0066] In actual applications, the system first predicts the first sample behavior to be taken in each state through the initial behavior model (Q table) based on the sample state data in the sample state set. For example, if the current state is that the player is at the entrance of the copy, the health is full, and the equipment is good, the Q table will predict the optimal behavior in this case based on the existing data, which may be "move forward 10 meters." The system will execute the behavior in the target copy based on the predicted first sample behavior. After execution, the system will record the player's new state information (first updated sample state data), such as the player's position after moving, health changes, etc., as well as the player's rewards or penalties for performing this behavior (first sample incentive data), such as experience points increase, health points decrease, etc.

[0067] Subsequently, the system updates the initial behavior model (Q table) using the first updated sample state data and the first sample incentive data. Specifically, the system adjusts the Q value of the corresponding state-action pair in the Q table according to the update formula of the Q-learning algorithm. The update formula is: new Q value = old Q value + learning rate * (immediate reward + discount factor * maximum Q value in the next state - old Q value). In this way, the Q table is continuously updated to gradually reflect the real benefits of performing specific actions in specific states, thereby optimizing the subsequent decision-making process.

[0068] In a specific embodiment of the present invention, it is assumed that the target copy is "Dark Forest". Based on a state data in the sample state set, the system predicts the optimal behavior to be "move forward 10 meters" through the Q table, such as the player is at the entrance of the forest, has full health and good equipment. The system executes this behavior in the "Dark Forest" copy. After the player moves, he encounters a monster. After the battle, the player's health is reduced by 5 points, but 10 experience points are obtained. The system records the new state information (the position of the player after moving, the remaining health, etc.) and the reward information (experience value increase, health value decrease). Then, the system updates the Q value of the corresponding state-action pair in the Q table based on this information, and calculates it using the update formula of the Q-learning algorithm. For example, assuming that the old Q value is 0, the learning rate is 0.1, the discount factor is 0.9, the maximum Q value in the next state is 5, and the immediate reward is 10-5=5, then the new Q value = 0+0.1*(5+0.9*5-0)=1.45. Through such an update process, the Q table gradually becomes more accurate and can better guide the behavior of the player in the copy.

[0069] Furthermore, based on each sample state data in the sample state set, a corresponding first sample behavior is predicted through an initial behavior model, including: for the first sample state data, based on the first sample behavior set, determining a behavior probability set corresponding to the first sample state data, wherein the first sample state data is any sample state data in the sample state set; based on the sample incentive set, obtaining a behavior value corresponding to each behavior data in the behavior probability set; calculating a probability value set based on the behavior value and the behavior probability set, and loading the probability value set into the initial behavior model; based on each sample state data in the sample state set, predicting the corresponding first sample behavior through the probability value set in the initial behavior model.

[0070] In actual applications, the system first predicts the corresponding first sample behavior for each sample state data in the sample state set through the initial behavior model. Specifically, the system determines the behavior probability set corresponding to each state based on the sample behavior set for each sample state data. The behavior probability set represents the probability of each behavior being executed in this state. These probabilities can be determined based on empirical values ​​or through statistical analysis.

[0071] Next, the system obtains the behavior value corresponding to each behavior data in the behavior probability set based on the sample incentive set. The behavior value reflects the immediate reward or punishment obtained after performing a certain behavior. For example, the system can calculate the average reward or punishment for each behavior in a specific state by statistically analyzing the existing sample data, thereby assigning a behavior value to each behavior.

[0072] Then, the system calculates the probability value set based on the behavior value and the behavior probability set. Specifically, the system multiplies the probability of each behavior by its corresponding behavior value to obtain the probability value of each behavior. These probability values ​​constitute the probability value set. The system loads these probability values ​​into the initial behavior model, which is usually a Q table, where rows represent states, columns represent behaviors, and the value of each cell represents the expected reward value for performing a specific behavior in a specific state.

[0073] Finally, based on the sample state data in the sample state set, the system predicts the corresponding first sample behavior through the probability value set in the initial behavior model. The system searches the Q table for the behavior probability value corresponding to the current state and selects the behavior with the highest probability value as the predicted first sample behavior. For example, assuming that the current state is "at the entrance of the dungeon, full health, and good equipment", the corresponding probability values ​​in the Q table are 0 for "moving forward 10 meters" and 2 for "using fireball to attack monsters", then the system will select "using fireball to attack monsters" as the predicted first sample behavior.

[0074] For the step-determined behavior probability set, one implementation method is to calculate the frequency of occurrence of each behavior in a specific state by statistically analyzing the existing sample data. Another implementation method is to manually assign an initial probability to each behavior based on expert experience or preset rules. In either case, the system needs to ensure the rationality and consistency of the initial probability in order to accurately reflect the player's behavior in the copy.

[0075] In a specific embodiment of the present invention, assuming that the target instance is "Dark Forest", the system obtains 1,000 records of successful clearance from the database, forming a sample state set, a sample behavior set, and a sample incentive data set. The system first determines the behavior probability set corresponding to a state data in the sample state set, such as "at the instance entrance, full health, good equipment", based on the sample behavior set. For example, the system statistically finds that the probability of "moving forward 10 meters" is 0.6, and the probability of "using fireball to attack monsters" is 0.4.

[0076] Next, the system obtains the behavior value corresponding to each behavior data in the behavior probability set based on the sample incentive set. For example, the system finds that the average reward for "moving forward 10 meters" is 0, and the average reward for "using fireball to attack monsters" is 5.

[0077] Then, the system calculates the probability value set based on the behavior value and the behavior probability set. For example, the probability value of "moving forward 10 meters" is 0.6*0=0, and the probability value of "using fireball to attack monsters" is 0.4*5=2. The system combines these probability values ​​into a probability value set.

[0078] The system loads the calculated probability value set into the initial behavior model. Assuming that the initial behavior model is a Q table, the system fills the values ​​in the probability value set into the cells of the corresponding state-behavior pairs in the Q table. For example, for the state "at the entrance of the dungeon, full health, good equipment", the value of "move forward 10 meters" in the Q table is 0, and the value of "use fireball to attack monsters" is 2.

[0079] Finally, the system predicts the corresponding first sample behavior based on the sample state data in the sample state set through the probability value set in the initial behavior model. For example, assuming the current state is "at the entrance of the dungeon, full health, good equipment", the corresponding probability values ​​in the Q table are "move forward 10 meters" is 0, and "use fireball to attack monsters" is 2, then the system will choose "use fireball to attack monsters" as the first sample behavior predicted.

[0080] Based on this, by predicting the corresponding first sample behavior through the initial behavior model based on the sample state data in the sample state set, the system can more accurately guide the player's behavior in the copy. This method not only improves the intelligence level of copy automation, but also makes more reasonable decisions based on the player's current state and historical behavior data, thereby improving the player's gaming experience. In addition, by combining behavior probability and behavior value, the system can more comprehensively evaluate the comprehensive value of each behavior, making the model more accurate and reliable in practical applications.

[0081] Furthermore, a first sample behavior is executed in the target copy to obtain corresponding first updated sample state data and first sample incentive data, and the initial behavior model is updated using the first updated sample state data and the first sample incentive data, including: executing the first sample behavior in the target copy to obtain corresponding first updated sample state data and first sample incentive data; calculating subsequent state return parameters based on the updated sample state data and the sample incentive data; obtaining preset correction parameters, and updating the initial behavior model based on the preset correction parameters, subsequent state return parameters, the sample state data and the first sample behavior.

[0082] Among them, the first sample behavior refers to the behavior that should be taken in a specific state according to the prediction of the initial behavior model. The first updated sample state data refers to the new state information of the player after executing the first sample behavior. The first sample incentive data refers to the reward or punishment obtained by the player after executing the first sample behavior. The initial behavior model refers to a model used to store the expected reward value that can be obtained by performing a certain action in a given state, usually expressed as a Q table. The subsequent state reward parameter refers to the expected future return calculated based on the new state and reward data. The preset correction parameter is a parameter used to adjust the Q value update formula, such as the learning rate and discount factor.

[0083] In actual applications, the system first executes the first sample behavior in the target copy and obtains the corresponding first updated sample state data and first sample incentive data. Specifically, the system executes the first sample behavior in the target copy according to the first sample behavior predicted by the initial behavior model. For example, assuming that the behavior predicted by the system is "using fireball to attack monsters", the system will control the player character to perform this behavior and record the results after execution, including new state information (such as player position, health value, etc.) and reward information (such as experience points obtained, health value changes, etc.).

[0084] Next, the system calculates the subsequent state reward parameter based on the updated sample state data and sample incentive data. The subsequent state reward parameter reflects the total reward that may be obtained in the future after executing the current behavior. The system can use a discount factor to calculate future rewards, for example, the subsequent state reward parameter = immediate reward + discount factor * maximum Q value in the new state. The discount factor is used to balance the importance of immediate rewards and future rewards.

[0085] Then, the system obtains the preset correction parameters, and updates the initial behavior model based on the preset correction parameters, subsequent state reward parameters, sample state data and the first sample behavior. The preset correction parameters usually include learning rate and discount factor. The learning rate determines the degree of influence of new information on old information, and the discount factor determines the weight of future rewards. The system uses the update formula of the Q-learning algorithm to update the values ​​in the Q table. The specific formula is: new Q value = old Q value + learning rate * (immediate reward + discount factor * maximum Q value in the new state - old Q value).

[0086] For the step calculation of the subsequent state reward parameters, one implementation method is to use a discount factor to calculate the present value of future rewards. Another implementation method is to use a dynamic programming method to iteratively calculate the optimal value function for each state. In either case, the system needs to ensure the accuracy and rationality of the calculation results in order to effectively update the Q table.

[0087] In a specific embodiment of the present invention, assuming that the target instance is "Dark Forest", the behavior predicted by the system based on the initial behavior model is "using fireball to attack monsters". The system executes this behavior in the "Dark Forest" instance and records the results after execution, including new state information (such as player position, health value, etc.) and reward information (such as experience points obtained, health value changes, etc.). Assume that the new state is "near the monster, health value reduced by 5 points", and the reward information is "10 experience points obtained".

[0088] Next, the system calculates the subsequent state reward parameter based on the updated sample state data and sample incentive data. Assuming the discount factor is 0.9, the maximum Q value in the new state is 50, and the immediate reward is 10, then the subsequent state reward parameter = 10 + 0.9 * 50 = 55.

[0089] Then, the system obtains the preset correction parameters, assumes the learning rate is 0.1, and updates the initial behavior model based on the preset correction parameters, subsequent state reward parameters, sample state data, and the first sample behavior. Assuming the old Q value is 20, the system uses the update formula of the Q-learning algorithm: new Q value = 20 + 0.1 * (55-20) = 23.5. The system fills the new Q value into the cell of the corresponding state-behavior pair in the Q table.

[0090] Based on this, by executing the first sample behavior in the target copy, obtaining the corresponding first updated sample state data and first sample incentive data, and using these data to update the initial behavior model, the system can gradually optimize the behavior model to more accurately predict the expected reward for performing a specific behavior in a specific state. This method not only improves the intelligence level of copy automation, but also dynamically adjusts the model based on feedback in the actual game environment, making the model more accurate and reliable in actual applications. In addition, by continuously updating and optimizing the Q table, the system can better adapt to different players and copies and enhance the player's gaming experience.

[0091] Furthermore, after obtaining the preset correction parameters and updating the initial behavior model based on the preset correction parameters, subsequent state reward parameters, sample state data and sample behavior, it also includes: obtaining the state value corresponding to the updated sample state data, and determining whether the state value is the terminal state value; if not, predicting the second sample behavior corresponding to the updated sample state data through the updated initial behavior model, executing the second sample behavior in the target copy, obtaining the corresponding second updated sample state data and second sample incentive data, and using the second updated sample state data and the second sample incentive data to update the initial behavior model; if so, recording the number of updates.

[0092] In actual applications, the system first obtains the state value corresponding to the updated sample state data. The state value is used to describe the completion progress of the target copy and reflects the overall performance of the player in a specific state. The system can obtain the state value by looking up the Q table or the predefined state value table.

[0093] Next, the system determines whether the state value is a terminal state value. The terminal state value indicates the state of the copy ending or the task completion. The system can predefine a terminal state value, such as 100, indicating that the player has successfully completed the copy. If the state value is not a terminal state value, the system predicts the second sample behavior corresponding to the updated sample state data through the updated initial behavior model. The system searches the Q table for the behavior probability value corresponding to the updated sample state data, and selects the behavior with the highest probability value as the predicted second sample behavior.

[0094] The system executes the predicted second sample behavior in the target copy and obtains the corresponding second updated sample state data and second sample incentive data. For example, assuming that the behavior predicted by the system is "move forward 10 meters", the system will control the player character to perform this behavior and record the results after execution, including new state information (such as player position, health value, etc.) and reward information (such as experience points obtained, health value changes, etc.).

[0095] Then, the system updates the initial behavior model using the second updated sample state data and the second sample incentive data. Based on the second updated sample state data and the second sample incentive data, the system calculates the subsequent state reward parameter and updates the value in the Q table using the update formula of the Q-learning algorithm. The specific formula is: New Q value = old Q value + learning rate * (immediate reward + discount factor * maximum Q value in the new state - old Q value).

[0096] If the state value is the terminal state value, the system records the number of updates. The number of updates is used to record the number of updates performed by the system during the training process, so as to facilitate subsequent analysis and evaluation of the model's performance.

[0097] For the step of obtaining the state value corresponding to the updated sample state data, one implementation method is to obtain it by looking up the Q table or the predefined state value table. Another implementation method is to calculate the average value of each state by statistically analyzing the existing sample data. In either case, the system needs to ensure the rationality and consistency of the state value in order to accurately determine whether the terminal state has been reached.

[0098] In a specific embodiment of the present invention, assuming that the target instance is "Dark Forest", after updating the initial behavior model, the system obtains the state value of the current state "near the monster, health value reduced by 5 points". Assuming that the state value is 80, the system determines that this is not the terminal state value (the predefined terminal state value is 100).

[0099] The system predicts the second sample behavior corresponding to the current state through the updated initial behavior model. Assuming that the system finds the highest probability value of "moving forward 10 meters" in the Q table, the behavior predicted by the system is "moving forward 10 meters". The system executes this behavior in the "Dark Forest" copy and records the results after execution, including new state information (such as player position, health points, etc.) and reward information (such as experience points gained, health point changes, etc.). Assuming that the new state is "approaching the end of the copy, health points reduced by 10 points", the reward information is "gaining 20 experience points".

[0100] The system updates the initial behavior model using the new state and reward data. Assuming the discount factor is 0.9, the maximum Q value in the new state is 70, the immediate reward is 20, and the old Q value is 23.5, the system uses the Q-learning algorithm's update formula: New Q value = 23.5 + 0.1 * (20 + 0.9 * 70 - 23.5) = 35.65. The system fills the new Q value into the cell of the corresponding state-behavior pair in the Q table.

[0101] Assuming the new state value is 95, the system again determines that this is not the final state value and continues the above process. If the final state value reaches 100, the system records the number of updates, indicating that the instance has been successfully cleared.

[0102] Based on this, after obtaining the preset correction parameters, updating the initial behavior model based on the preset correction parameters, subsequent state return parameters, sample state data and sample behavior, the system can further process and optimize the model. This method not only improves the intelligence level of copy automation, but also dynamically adjusts the model according to feedback in the actual game environment, making the model more accurate and reliable in actual applications. In addition, by recording the number of updates, the system can evaluate the training effect of the model and further optimize the model performance.

[0103] Step 208: When the update stop condition is met, obtain the replica policy model.

[0104] The copy strategy model refers to the model that is finally obtained through the training process and can guide players to perform optimal behaviors in the copy. The update stop condition refers to stopping further updates of the model when specific criteria are met during the training process. For example, a fixed number of training rounds can be set, such as 1,000 rounds, or the update can be stopped when the performance indicator of the model (such as average reward) reaches or exceeds a certain threshold.

[0105] In practical applications, the system first defines the update stop condition. One of the common conditions is to set a fixed number of training rounds, such as 1000 rounds. This means that the system will perform 1000 rounds of training. Each round of training includes selecting a state from the sample state set, predicting the corresponding behavior through the initial behavior model (Q table), executing the behavior, recording the new state and reward, and updating the Q table. In each round of training, the system will continuously optimize the Q table so that it can better predict the expected reward for performing a specific action in a specific state.

[0106] When the system completes 1000 rounds of training, the update stop condition is reached. At this point, the system stops updating the Q table further and uses the current Q table as the final dungeon strategy model. This model contains the expected reward values ​​for performing different actions in various states, which can be used to guide the player's behavior in the dungeon and help the player complete the dungeon challenge more efficiently.

[0107] In a specific embodiment of the present invention, it is assumed that the target copy is "Dark Forest". The system sets the training rounds to 1000 rounds. In each round of training, the system randomly selects a state from the sample state set and predicts the optimal behavior through the Q table, such as "move forward 10 meters". The system executes this behavior in the "Dark Forest" copy, records the new state information (such as the position of the player after moving, the change of health value, etc.) and reward information (such as experience value increase, health value decrease, etc.). Then, the system updates the Q value of the corresponding state-action pair in the Q table based on this information, and calculates it using the update formula of the Q-learning algorithm. For example, assuming that the old Q value is 0, the learning rate is 0.1, the discount factor is 0.9, the maximum Q value in the next state is 5, and the immediate reward is 10-5=5, then the new Q value = 0+0.1*(5+0.9*5-0)=1.45. The system repeats this process until 1000 rounds of training are completed.

[0108] Further, when the update stop condition is met, the replica strategy model is obtained, including: when the number of updates reaches a preset update threshold, the replica strategy model is obtained.

[0109] In actual applications, the system obtains the replica strategy model when the update stop condition is met. Specifically, the system records the number of updates each time the initial behavior model is updated. When the number of updates reaches the preset update threshold, the system stops further updating the initial behavior model and uses the current initial behavior model as the final replica strategy model. The preset update threshold can be set according to actual needs, for example, it can be set to 1000 updates.

[0110] In a specific embodiment of the present invention, it is assumed that the target replica is "Dark Forest" and the preset update threshold set by the system is 1000 times. During the training process, the system records the current number of updates after each update of the initial behavior model. It is assumed that after the 999th update, the system continues to perform an update and the number of updates reaches 1000 times. The system checks the number of updates and finds that the preset update threshold has been reached, so it stops further updating the initial behavior model.

[0111] Based on this, by obtaining the replica strategy model when the update stop conditions are met, the system can ensure that the training is completed within a limited time and a relatively stable model is obtained. This method not only ensures the controllability of the training process, but also obtains a model with good performance in a shorter time, thereby improving the player's gaming experience. In addition, through fixed rounds of training, the system can avoid overfitting, ensure that the model has good generalization ability, and can perform stably in different game environments.

[0112] An embodiment of the present invention realizes obtaining a sample data set corresponding to a target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; constructing an initial behavior model based on the sample state set, the sample behavior set, and the sample incentive set; predicting the corresponding first sample behavior through the initial behavior model based on each sample state data in the sample state set, executing the first sample behavior in the target replica, obtaining the corresponding first updated sample state data and the first sample incentive data, and updating the initial behavior model using the first updated sample state data and the first sample incentive data; obtaining a replica strategy model when the update stop condition is met. An initial behavior model is constructed by collecting and analyzing sample data of the target replica, including the sample state set, the sample behavior set, and the sample incentive set. Based on this model, the system can predict and execute the behavior that best matches the current state, and update the model according to the execution result. This process is iterated repeatedly until the model reaches the optimal state, and finally generates a personalized replica strategy model. Taking into account the individual differences of players, the strategy can also be dynamically adjusted to ensure that each player can obtain the best gaming experience, which can significantly improve the automation level of the game and the player satisfaction, so that players can efficiently advance the game progress in fragmented time and enjoy a smooth gaming experience.

[0113] See also Figure 3 , Figure 3 It is a flowchart of a game copy execution method provided by an embodiment of the present invention, which specifically includes the following steps.

[0114] Step 302: In response to the copy automatic execution instruction issued by the front-end user, the copy identifier of the target copy and the user identifier of the front-end user are determined.

[0115] The copy ID is used to uniquely identify the copy that the player wants to automatically execute, and the user ID is used to confirm the identity of the player who executes the copy automation.

[0116] In actual applications, when a player triggers the automatic copy execution function through the game interface, the system first receives and parses the player's request. The request usually contains some necessary information, such as the copy name or ID (copy identifier) ​​selected by the player and the player's account information (user identifier). By parsing this information, the system determines the specific copy that the player wants to execute automatically and the identity of the player who performs the operation. Specifically, the system extracts the copy identifier and user identifier from the request, and then verifies the legitimacy of this information. For example, the system checks whether the copy ID exists in the database and whether the user ID corresponds to an active player account. After verification, the system will record the relevant information of this request, including the request time, copy identifier, user identifier, etc., for subsequent debugging and auditing.

[0117] For step parsing requests, one implementation method is to use an HTTP request parsing library to parse the JSON format request sent by the front-end user into the system's internal data structure. Another implementation method is to use a custom protocol between the game client and the server to extract the required information by parsing binary data packets. In either case, the system needs to ensure that the extracted information is accurate and can respond quickly to the player's request.

[0118] In a specific embodiment of the present invention, it is assumed that the player "Brave Warrior" chooses to automatically challenge the "Dark Forest" copy in an MMORPG game. The player initiates a request through the "Auto Challenge" button on the game interface. After receiving the request, the system parses the copy identifier "1001" (the ID of "Dark Forest") and the user identifier "12345" (the user ID of "Brave Warrior"). The system verifies the validity of the two identifiers and confirms that "1001" does correspond to an existing copy and "12345" corresponds to an active player account. After the verification is passed, the system records the log of this request, including the request time of 22:30 on November 1, 2024, the copy identifier "1001", and the user identifier "12345". The system also checks the current status of the player "Brave Warrior", such as level, equipment, health, etc., to ensure that the player has the conditions to challenge the copy.

[0119] Step 304: Obtain a replica policy model corresponding to the target replica based on the replica identifier, wherein the replica policy model is obtained based on a policy model generation method.

[0120] The instance identifier is used to uniquely identify the instance that the player wants to automatically execute, and the instance strategy model is generated based on the training data of a specific instance to guide the player's behavior in the instance. The strategy model generation method refers to the model generation process obtained through training with a machine learning algorithm (such as Q-learning).

[0121] In actual applications, when the system determines the replica ID selected by the player, it will obtain the replica strategy model of the corresponding target replica from the database based on the ID. Specifically, the system first extracts the unique identifier of the replica from the replica ID, such as "1001" (corresponding to the "Dark Forest" replica). Then, the system accesses the database that stores the replica strategy model to find the strategy model that matches the replica ID. The replica strategy model is usually a Q table that contains the expected reward value for performing a specific action in a specific state. The system quickly locates the corresponding Q table through the replica ID and loads it into memory for subsequent use.

[0122] In a specific embodiment of the present invention, it is assumed that the player "Brave Warrior" chooses to automatically challenge the "Dark Forest" copy in an MMORPG game. After receiving the request, the system parses the copy identification "1001" (the ID of "Dark Forest"). The system accesses the database storing the copy strategy model to find the strategy model that matches "1001". It is assumed that there is already a Q table obtained after 1000 rounds of training in the database, and the Q table contains the expected reward values ​​for performing different actions in various states in the "Dark Forest" copy. The system loads this Q table into the memory and prepares to use it to guide the behavior of the player "Brave Warrior" in the "Dark Forest" copy.

[0123] Step 306: Obtain the state information of the target replica based on the user identifier, and input the state information into the replica strategy model to predict and obtain an action instruction.

[0124] In actual applications, after the system determines the player's user ID, it will obtain the state information of the target copy from the database based on the ID. Specifically, the system first extracts the player's unique identifier from the user ID, such as "12345" (corresponding to the player "Brave Warrior"). Then, the system accesses the database that stores the player's state information and searches for state information that matches the user ID. This state information may include the player's position in the copy, health points, equipment status, current task progress, etc. The system organizes this state information into a structured data format, such as a vector or dictionary.

[0125] In a specific embodiment of the present invention, following the above example, the system inputs the sorted state information into the copy strategy model of the "Dark Forest" copy. Assuming that the copy strategy model is a Q table, the system searches the Q table for the corresponding state-action pair based on the current state information, and selects the action with the highest expected reward value as the predicted action instruction. For example, the Q table may show that in the current state, the optimal behavior is "move forward 10 meters." The system passes this action instruction to the game engine, controlling the player "Brave Warrior" to perform the behavior.

[0126] Step 308: Execute the target copy based on the action instruction.

[0127] An embodiment of the present invention implements the following steps: responding to a copy automatic execution instruction issued by a front-end user, determining the copy identifier of a target copy and the user identifier of a front-end user; obtaining a copy strategy model corresponding to the target copy based on the copy identifier, wherein the copy strategy model is obtained based on a strategy model generation method; obtaining the state information of the target copy based on the user identifier, and inputting the state information into the copy strategy model to predict an action instruction; and executing the target copy based on the action instruction. By responding to a copy automatic execution instruction issued by a front-end user, determining the identifier and user identifier of the target copy, and obtaining a copy strategy model corresponding to the target copy. The model is based on an advanced strategy generation method and can adapt to the specific circumstances of different players. Subsequently, the system obtains the state information of the target copy based on the user identifier, and inputs this information into the copy strategy model to predict the optimal action instruction. Finally, the system executes the target copy according to the predicted action instruction, realizing an efficient and personalized copy automatic execution function. This not only reduces the operating burden of the player, but also improves the game progress in fragmented time, significantly improving the player's gaming experience.

[0128] The following combination Figure 4 Taking the application of the strategy model generation method provided by the present invention in the automatic hanging AI of the game copy as an example, the strategy model generation method is further explained. Figure 4 It is a process flow chart of a strategy model generation method provided by an embodiment of the present invention, which specifically includes the following steps.

[0129] Step 402: Collect player data and define the state, action, and reward of the Q function.

[0130] Specifically, we first need to collect sample data of the dungeon process and record the various actions of the players in the dungeon, including the coordinates of the moving position, the skills released, the dialogue information, the click dialogue events, etc. At the same time, we need to record the data of each frame in the dungeon, including the location information of the NPC, the status information of the NPC, the status information of the player, the channel information, the current progress of the dungeon, etc. In addition, we also need to record the benefits or penalties of the players after each action, including various Buff bonuses, item benefits, in-game penalties, level advancement, etc. By processing these data, we can construct the State, Action, and Reward sets required for Q-Learning reinforcement learning.

[0131] Step 404: Establish a Q function model.

[0132] Specifically, the collected Reward data is used to assign an initial value to each action in the Action set. For example, the value of a normal move is 0, and the value of consuming an item is -5. At the same time, a value is assigned to each state in the State set. These state values ​​are evaluated based on information such as the copy progress and level settings. The closer to the end level, the higher the state value. With this information, the Q-table of Q-Learning reinforcement learning is established and initialized. The Q-table defines the reward value R for executing action A in state S. R represents the benefit of executing action A in state S.

[0133] Step 406: Start training.

[0134] Specifically, the training process officially begins. At this stage, the system will gradually optimize the Q value in the Q table based on the initialized Q table and the collected sample data to find the optimal strategy model.

[0135] Step 408: Execute actions using a greedy strategy.

[0136] Specifically, during the training process, a greedy strategy is used to execute each action. In order to make the strategy more consistent with the actual choices of the players, the system will count the different actions performed by external network players in a specific state and their frequency of occurrence to obtain the player's action probability P. Then, the action value R of different actions is multiplied by the player's action probability P to obtain the probability M of each action executed during training. This can better simulate the behavior pattern of real players. The specific formula is as follows:

[0137] Step 410: Update the formula Q value using the target value and the Q learning model.

[0138] Specifically, after each action is executed, the system will obtain a new state and reward. The Q value is updated using the Q learning model update formula (1). Formula (1) is as follows: [ ](1) in, The estimated value of taking an action in the current state.

[0139] α is the learning rate, which determines the learning speed of new knowledge. It is between 0 and 1 and is usually small to ensure convergence.

[0140] To perform actions Instant rewards received immediately afterwards.

[0141] γ is a discount factor, which also ranges from 0 to 1. It reflects the impact of future rewards on current rewards. If γ is close to 1, long-term rewards are more important; if it is close to 0, only short-term rewards are concerned.

[0142] Indicates that in the next state The maximum state reward of all possible actions in . This is done by traversing all possible actions And select the action with the highest Q value to determine.

[0143] Step 412: Whether the maximum number of rounds has been reached.

[0144] Specifically, the system will check whether the preset maximum number of rounds has been reached. If the maximum number of rounds has not been reached, steps 406-410 will continue to be executed for the next round of training. If the maximum number of rounds has been reached, step 414 will be executed.

[0145] Step 414: configure the trained Q function into the policy model.

[0146] Specifically, after the training is completed, the trained Q function is configured into the strategy model. Define the intelligent agent and load the trained Q table. The agent can interact with the player's copy environment, input the current state information, return the player's current highest value action, and guide the player to complete the copy. In addition, the initial action value and state value can be adjusted according to the actual test results to further optimize the player's action strategy until the expected effect is achieved.

[0147] Corresponding to the above method embodiment, the present invention also provides a strategy model generation device embodiment, Figure 5 FIG. 1 is a schematic diagram of a strategy model generation device provided by an embodiment of the present invention. Figure 5 As shown, the device comprises: A first acquisition module 502 is configured to acquire a sample data set corresponding to a target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; Optionally, the first acquisition module 502 is further configured to acquire a completed copy data set based on the copy identifier of the target copy, wherein the completed copy data set is the copy data of the target copy for clearance; parse the completed copy data set to obtain behavior data representing user behavior as a sample behavior set, state data representing the target copy during the clearance process as a sample state set, and feedback information corresponding to the behavior data as a sample incentive set; and use the sample state set, sample behavior set, and sample incentive set as the sample data set corresponding to the target copy.

[0148] The construction module 504 is configured to construct an initial behavior model based on the sample state set, the sample behavior set and the sample stimulus set.

[0149] Optionally, the construction module 504 is further configured to determine the behavior value corresponding to each sample behavior data in the sample behavior set based on the sample incentive set; determine the state value corresponding to each sample state data in the sample state set based on the sample incentive set, wherein the state value is used to describe the completion progress of the target copy; and construct an initial behavior model based on the sample state set, the sample behavior set, the behavior value and the state value.

[0150] The updating module 506 is configured to predict the corresponding first sample behavior through the initial behavior model based on each sample state data in the sample state set, execute the first sample behavior in the target replica, obtain the corresponding first updated sample state data and first sample incentive data, and update the initial behavior model using the first updated sample state data and the first sample incentive data; Optionally, the update module 506 is further configured to determine, for the first sample state data, based on the first sample behavior set, a behavior probability set corresponding to the first sample state data, wherein the first sample state data is any sample state data in the sample state set; based on the sample incentive set, obtain a behavior value corresponding to each behavior data in the behavior probability set; calculate a probability value set based on the behavior value and the behavior probability set, and load the probability value set into the initial behavior model; based on each sample state data in the sample state set, predict the corresponding first sample behavior through the probability value set in the initial behavior model.

[0151] Optionally, the update module 506 is further configured to execute a first sample behavior in the target copy to obtain corresponding first updated sample state data and first sample incentive data; calculate subsequent state return parameters based on the updated sample state data and the sample incentive data; obtain preset correction parameters, and update the initial behavior model based on the preset correction parameters, subsequent state return parameters, sample state data and the first sample behavior.

[0152] Optionally, the update module 506 is further configured to obtain a state value corresponding to the updated sample state data, and determine whether the state value is a terminal state value; if not, predict a second sample behavior corresponding to the updated sample state data through the updated initial behavior model, execute the second sample behavior in the target copy, obtain the corresponding second updated sample state data and second sample incentive data, and use the second updated sample state data and the second sample incentive data to update the initial behavior model; if so, record the number of updates.

[0153] The stop module 508 is configured to obtain a replica policy model when the update stop condition is met.

[0154] Optionally, the stopping module 508 is further configured to obtain a replica strategy model when the number of updates reaches a preset update threshold.

[0155] Applied to the strategy model generation device, the sample data set of the target copy is obtained through the first acquisition module 502, including the sample state set, the sample behavior set and the sample incentive set, and the copy data set is parsed to extract detailed behavior, state and feedback information. The construction module 504 constructs an initial behavior model based on these data sets to ensure that the model can learn the strategy of taking the best action in different states. Through the collaborative work of these two modules, the device can efficiently generate a personalized copy strategy model, improve the automation level of the game and the game experience of the player.

[0156] The above is a schematic scheme of a strategy model generation device of this embodiment. It should be noted that the technical scheme of the strategy model generation device and the technical scheme of the strategy model generation method described above belong to the same concept, and the details not described in detail in the technical scheme of the strategy model generation device can be found in the description of the technical scheme of the strategy model generation method described above.

[0157] Corresponding to the above method embodiment, the present invention also provides a game copy execution device embodiment, Figure 6 1 is a schematic diagram of the structure of a game copy execution device provided by an embodiment of the present invention. Figure 6 As shown, the device comprises: The determination module 602 is configured to determine the replica identifier of the target replica and the user identifier of the front-end user in response to the replica automatic execution instruction issued by the front-end user; The second acquisition module 604 is configured to acquire a replica policy model corresponding to the target replica based on the replica identifier, wherein the replica policy model is obtained based on a policy model generation method; An input module 606 is configured to obtain state information of a target replica based on a user identifier, and input the state information into a replica strategy model to predict an action instruction; The execution module 608 is configured to execute the target copy based on the action instruction.

[0158] Applied to the game copy execution device, the determination module 602 responds to the copy automatic execution instruction issued by the front-end user to determine the copy identifier and user identifier of the target copy; the second acquisition module 604 obtains the corresponding copy strategy model based on the copy identifier; the input module 606 obtains the state information of the target copy based on the user identifier, and inputs the state information into the copy strategy model to predict the action instruction; the execution module 608 executes the target copy based on the predicted action instruction. Through the collaborative work of these modules, the device realizes efficient and personalized copy automatic execution, which significantly improves the game experience of players.

[0159] The above is a schematic scheme of a game copy execution device of this embodiment. It should be noted that the technical scheme of the game copy execution device and the technical scheme of the game copy execution method described above belong to the same concept, and the details not described in detail in the technical scheme of the game copy execution device can be referred to the description of the technical scheme of the game copy execution method described above.

[0160] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present invention is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0161] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0162] In one embodiment of the present invention, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present invention. Those skilled in the art may add or replace other components as needed.

[0163] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0164] The processor 720 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned strategy model generation method and game copy execution method.

[0165] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the strategy model generation method and the game copy execution method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the strategy model generation method and the game copy execution method embodiments.

[0166] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned strategy model generation method and game copy execution method when executed by a processor.

[0167] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the strategy model generation method and the game copy execution method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the strategy model generation method and the game copy execution method embodiments.

[0168] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned strategy model generation method and game copy execution method when executed by a processor.

[0169] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned strategy model generation method and game copy execution method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned strategy model generation method and game copy execution method.

[0170] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0172] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present invention.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present invention. The present invention selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A strategy model generation method, characterized in that: include: Obtaining a sample data set corresponding to the target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; Constructing an initial behavior model based on the sample state set, the sample behavior set and the sample stimulus set; Based on each sample state data in the sample state set, predict a corresponding first sample behavior through the initial behavior model, execute the first sample behavior in the target copy, obtain corresponding first updated sample state data and first sample incentive data, and update the initial behavior model using the first updated sample state data and the first sample incentive data; When the update stop condition is met, the replica policy model is obtained.

2. The method according to claim 1, characterized in that The step of obtaining a sample data set corresponding to the target replica includes: Based on the replica identifier of the target replica, obtaining a completed replica data set, wherein the completed replica data set is replica data that passes the target replica; Parsing the completed copy data set, obtaining behavior data representing user behavior as a sample behavior set, state data representing the target copy during the clearance process as a sample state set, and feedback information corresponding to the behavior data as a sample incentive set; The sample state set, the sample behavior set and the sample incentive set are used as sample data sets corresponding to the target copy.

3. The method according to claim 1, characterized in that The constructing the initial behavior model based on the sample state data set, the sample behavior data set and the sample incentive data set includes: Based on the sample incentive set, determining the behavior value corresponding to each sample behavior data in the sample behavior set; Based on the sample incentive set, determining a state value corresponding to each sample state data in the sample state set, wherein the state value is used to describe the completion progress of the target copy; An initial behavior model is constructed based on the sample state set, the sample behavior set, the behavior value and the state value.

4. The method according to claim 1, characterized in that: The predicting the corresponding first sample behavior by the initial behavior model based on each sample state data in the sample state set includes: For first sample state data, based on the first sample behavior set, determining a behavior probability set corresponding to the first sample state data, wherein the first sample state data is any sample state data in the sample state set; Based on the sample incentive set, obtaining the behavior value corresponding to each behavior data in the behavior probability set; Calculating a probability value set based on the behavior value and the behavior probability set, and loading the probability value set into the initial behavior model; Based on each sample state data in the sample state set, a corresponding first sample behavior is predicted by the probability value set in the initial behavior model.

5. The method according to claim 1, characterized in that The executing the first sample behavior in the target copy, obtaining corresponding first updated sample state data and first sample incentive data, and updating the initial behavior model by using the first updated sample state data and the first sample incentive data, comprises: said executing the first sample behavior in the target copy to obtain corresponding first updated sample state data and first sample incentive data; Calculating subsequent state reward parameters based on the updated sample state data and the sample incentive data; Obtain a preset correction parameter, and update the initial behavior model based on the preset correction parameter, the subsequent state reporting parameter, the sample state data, and the first sample behavior.

6. The method according to claim 5, characterized in that After obtaining the preset correction parameter and updating the initial behavior model based on the preset correction parameter, the subsequent state reporting parameter, the sample state data and the sample behavior, the method further includes: Obtaining a state value corresponding to the updated sample state data, and determining whether the state value is a terminal state value; If not, predicting a second sample behavior corresponding to the updated sample state data by using the updated initial behavior model, executing the second sample behavior in the target copy to obtain corresponding second updated sample state data and second sample stimulus data, and updating the initial behavior model by using the second updated sample state data and the second sample stimulus data; If yes, record the update times.

7. The method according to claim 6, characterized in that When the update stop condition is met, obtaining the replica strategy model includes: When the number of updates reaches a preset update threshold, a copy strategy model is obtained.

8. A method for executing a game copy, characterized in that: include: In response to a copy automatic execution instruction issued by a front-end user, determining a copy identifier of a target copy and a user identifier of the front-end user; Acquire a replica policy model corresponding to the target replica based on the replica identifier, wherein the replica policy model is obtained based on the policy model generation method according to any one of claims 1 to 7; Acquire the state information of the target replica based on the user identifier, and input the state information into the replica strategy model to predict and obtain an action instruction; The target copy is executed based on the action instruction.

9. A strategy model generation device, characterized in that: include: A first acquisition module is configured to acquire a sample data set corresponding to a target replica, wherein the sample data set includes a sample state set, a sample behavior set, and a sample incentive set; A construction module, configured to construct an initial behavior model based on the sample state set, the sample behavior set and the sample stimulus set; an updating module, configured to predict a corresponding first sample behavior through the initial behavior model based on each sample state data in the sample state set, execute the first sample behavior in the target copy, obtain corresponding first updated sample state data and first sample incentive data, and update the initial behavior model using the first updated sample state data and the first sample incentive data; The stop module is configured to obtain the replica strategy model when the update stop condition is met.

10. A game copy execution device, characterized in that: include: A determination module, configured to determine the copy identifier of the target copy and the user identifier of the front-end user in response to the copy automatic execution instruction issued by the front-end user; A second acquisition module is configured to acquire a replica policy model corresponding to the target replica based on the replica identifier, wherein the replica policy model is obtained based on the policy model generation method according to any one of claims 1 to 7; An input module is configured to obtain the state information of the target replica based on the user identifier, and input the state information into the replica strategy model to predict and obtain an action instruction; An execution module is configured to execute the target copy based on the action instruction.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the strategy model generation method described in any one of claims 1 to 7 and the game copy execution method described in claim 8 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer programs / instructions, which, when executed by a processor, implement the steps of the strategy model generation method described in any one of claims 1 to 7 and the game copy execution method described in claim 8.

13. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the steps of the strategy model generation method described in any one of claims 1 to 7 and the game copy execution method described in claim 8.

Images