Game strategy model training method and related equipment

Abstract

The invention discloses a game strategy model training method and related equipment, and the method comprises the steps: controlling the game actions of an intelligent agent in a target game through a game strategy model of the target game based on the game state of the intelligent agent; based on the game state and a historical game state, the state information entropy of the intelligent agent at the moment corresponding to the game state is calculated, and the historical game state is the state of the intelligent agent appearing before the game state in the current game; determining corresponding exploration reward information of the intelligent agent in the game state according to the state information entropy; and performing parameter updating on the game strategy model based on the exploration reward information and the game state and the game action when the intelligent agent executes the action each time to obtain a target game strategy model. According to the method, the accuracy of game action generation of the target game strategy model can be improved, and obstacles in the game can be identified more accurately, so that the obstacle avoidance capability of the intelligent agent is improved, and the clearance capability of the intelligent agent is enhanced.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method for training a game strategy model and related equipment. Background Technology

[0002] With the research and advancement of artificial intelligence technology, it has been studied and applied in many fields, such as smart homes, smart wearable devices, autonomous driving, self-driving cars, drones, robots, and game AI (Artificial Intelligence).

[0003] In game AI research, current methods for building game AI agents generally involve designing reasonable level-clearing rewards, enabling the agent to learn corresponding path completion strategies and achieve victory. However, for racing games, new levels are frequently released. Current training methods relying solely on level-clearing rewards result in agents with fixed and simplistic path completion strategies. This leads to poor performance in levels with varying obstacle placements and low accuracy in obstacle recognition, limiting the application of these agents in racing games. Summary of the Invention

[0004] This application provides a game strategy model training method and related equipment. The related equipment may include a game strategy model training device, electronic equipment, computer-readable storage medium, and computer program product. It can improve the accuracy of game action generation of the target game strategy model, more accurately identify obstacles in the game, thereby improving the obstacle avoidance ability of the intelligent agent and enhancing the ability of the intelligent agent to pass the level.

[0005] This application provides a method for training a game strategy model, including:

[0006] Based on the game state of the agent, the game strategy model of the target game is used to control the game actions of the agent in the target game. The game strategy model is used to determine the game actions that the agent should perform in different game states.

[0007] Based on the game state and the historical game state, calculate the state information entropy of the agent at the time corresponding to the game state. The historical game state is the state that the agent appeared in before the game state in the current game.

[0008] Based on the state information entropy, the exploration reward information corresponding to the agent in the game state is determined;

[0009] Based on the exploration reward information and the game state and actions of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0010] Accordingly, embodiments of this application provide a game strategy model training device, comprising:

[0011] The control unit is used to control the game actions of the agent in the target game based on the game state of the agent, using the game strategy model of the target game. The game strategy model is used to determine the game actions to be performed by the agent in different game states.

[0012] The computing unit is used to calculate the state information entropy of the agent at the time corresponding to the game state based on the game state and the historical game state, wherein the historical game state is the state of the agent in the current game before the game state.

[0013] The reward determination unit is used to determine the exploration reward information corresponding to the agent in the game state based on the state information entropy.

[0014] The update unit is used to update the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, so as to obtain the target game strategy model.

[0015] Optionally, in some embodiments of this application, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0016] The control unit may include a first determining subunit, a second determining subunit, and a control subunit, as follows:

[0017] The first determining subunit is used to input the game state of the agent into the game strategy model of the target game to determine at least one candidate game action of the agent;

[0018] The second determining subunit is used to determine the target game action to be performed by the agent from the candidate game actions by using the probability distribution information of the game strategy model;

[0019] A control subunit is used to control the agent to perform target game actions in the target game.

[0020] Optionally, in some embodiments of this application, the computing unit includes a sequence determination subunit, an encoding subunit, a classification subunit, and an entropy calculation subunit, as follows:

[0021] The sequence determination subunit is used to determine the historical state sequence corresponding to the game state, and the historical state sequence includes at least one historical game state corresponding to the game state and the game state.

[0022] The encoding subunit is used to perform state encoding processing on each state in the historical state sequence to obtain the state encoding information of each state in the historical state sequence;

[0023] The classification subunit is used to classify the state coding information of each state in order to determine the proportion of occurrence of each state under each preset state type.

[0024] The entropy calculation subunit is used to calculate the type information entropy of the historical state sequence based on the occurrence ratio information of each preset state type, so as to obtain the state information entropy of the agent at the corresponding moment of the game state.

[0025] Optionally, in some embodiments of this application, the action type of the intelligent agent's game actions includes movement action types;

[0026] Specifically, the encoding subunit can be used to obtain the position information of the agent in the target game under each state of the historical state sequence; and determine the state encoding information of each state in the historical state sequence based on the position information.

[0027] Optionally, in some embodiments of this application, the encoding subunit may be specifically used to perform state encoding processing on each state in the historical state sequence through a state encoding model to obtain state encoding information of each state in the historical state sequence.

[0028] The game strategy model training device may further include a state coding model training unit, as follows:

[0029] The state encoding model training unit is used to acquire training data, which includes at least one sample game state of the agent in the target game and the expected state encoding information corresponding to the sample game state; the sample game state is processed by the state encoding model to obtain the sample state encoding information of the sample game state; the parameters of the state encoding model are adjusted according to the sample state encoding information and the expected state encoding information to obtain the trained state encoding model.

[0030] Optionally, in some embodiments of this application, the game strategy model training device may further include an expected encoding determination unit, as follows:

[0031] The expected encoding determination unit is used to divide the preset encoding space of the game state of the intelligent agent according to the exploration ability requirement information of the intelligent agent, to obtain multiple state encoding spaces, each state encoding space corresponding to a state encoding information; and to determine the expected state encoding information corresponding to the sample game state according to the state encoding space in which the sample game state is located.

[0032] Optionally, in some embodiments of this application, the reward determination unit may include a third determination subunit and a fourth determination subunit, as follows:

[0033] The third determining subunit is used to determine the exploration reward information corresponding to the agent in the game state as the first exploration reward information when the state information entropy is greater than a preset threshold.

[0034] The fourth determining subunit is used to determine the exploration reward information corresponding to the agent in the game state as the second exploration reward information when the state information entropy is not greater than a preset threshold, wherein the second exploration reward information is less than the first exploration reward information.

[0035] Optionally, in some embodiments of this application, the reward determination unit may include a function acquisition subunit and a computation subunit, as follows:

[0036] The function acquisition subunit is used to acquire a preset incrementing function;

[0037] The computational subunit is used to process the state information entropy through the preset incrementing function to obtain the exploration reward information corresponding to the agent in the game state.

[0038] Optionally, in some embodiments of this application, the updating unit may include an association state determination subunit, an action reward / penalty determination subunit, and an updating subunit, as follows:

[0039] The associated state determination subunit is used to determine the associated game states corresponding to each game state of the agent in the current game.

[0040] The action reward and punishment determination subunit is used to determine the action reward and punishment information corresponding to each game state based on the exploration reward information corresponding to the game state and the exploration reward information corresponding to the associated game state.

[0041] The update subunit is used to update the parameters of the game strategy model based on the game state and game action of the agent each time it performs an action, as well as the action reward and punishment information corresponding to each game state of the agent in the current game, to obtain the target game strategy model.

[0042] Optionally, in some embodiments of this application, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0043] The update subunit can be specifically used to construct the action data sequence of the agent. The action data sequence includes various strategy data pairs arranged by execution time in the current game. The strategy data pair includes the game state and game action at the corresponding execution time. For each strategy data pair, a target game state that matches the strategy data pair in the game strategy model is determined. Based on the action reward and penalty information corresponding to the game state in the strategy data pair, the probability distribution information of the candidate game actions corresponding to the target game state is updated to obtain the target game strategy model.

[0044] Optionally, in some embodiments of this application, the updating unit may include a reward / penalty determination subunit, a parameter update subunit, a fifth determination subunit, and a return execution subunit, as follows:

[0045] The reward and punishment determination subunit is used to determine the action reward and punishment information corresponding to the current game state based on the exploration reward information corresponding to the current game state of the agent and the exploration reward information of the game state corresponding to the current game state after the agent performs the current game action;

[0046] The parameter update subunit is used to update the parameters of the game strategy model based on the action reward and punishment information of the current game state, the current game state, and the game actions performed by the agent in the current game state.

[0047] The fifth determining subunit is used to take the game state after the agent performs the game action as the new current game state;

[0048] The execution subunit is returned to control the agent's game actions in the target game based on the updated game strategy model and the agent's new current game state. The execution subunit then returns to the step of calculating the state information entropy of the agent at the corresponding moment in the game state based on the game state and historical game states, so as to obtain the target game strategy model.

[0049] Optionally, in some embodiments of this application, the updating unit may include a model updating subunit, an evaluation subunit, and a sixth determining subunit, as follows:

[0050] The model update subunit is used to update the parameters of the game strategy model based on the exploration reward information and the game state and game action of the agent each time it performs an action, so as to obtain the updated game strategy model.

[0051] The evaluation subunit is used to evaluate the obstacle avoidance ability of the agent based on the state information entropy corresponding to each game state in the target game.

[0052] The sixth determining subunit is used to determine the updated game strategy model as the target game strategy model when the updated game strategy model passes the evaluation.

[0053] Optionally, in some embodiments of this application, the game strategy model training device further includes an acquisition unit, which is used to acquire the game strategy model of the target game; specifically, the acquisition unit may include a model acquisition subunit, an action control subunit, and an initial model update subunit, as follows:

[0054] The model acquisition subunit is used to acquire the initial game strategy model of the target game. The initial game strategy model is used to determine the game actions that the agent should perform in different game states.

[0055] An action control subunit is used to control the agent's game actions in the target game based on the agent's game state in multiple games of the target game, using the initial game strategy model.

[0056] The initial model update subunit is used to update the parameters of the initial game strategy model based on the game state and game actions of the agent each time it performs an action that meets the preset level-clearing conditions, so as to obtain the game strategy model. The agent under the control of the game strategy model has the ability to clear the level.

[0057] Optionally, in some embodiments of this application, the initial model update subunit may be specifically used to calculate the completion rate of the agent in the target game based on the game result data of the agent in each game; when the completion rate does not meet the preset conditions, the parameters of the initial game strategy model are updated based on the game state and game actions of the agent that meets the preset completion conditions each time it performs an action, to obtain an updated initial game strategy model; and return to execute the step of controlling the agent's game actions in the target game based on the agent's game state in multiple games of the target game through the initial game strategy model, until the completion rate of the agent in the target game meets the preset conditions, to obtain the game strategy model.

[0058] Optionally, in some embodiments of this application, the game strategy model training device may further include a model application unit, as follows:

[0059] The model application unit is used to acquire the current game state data of the agent in the target game; input the current game state data of the agent into the target game strategy model, so as to control the agent's game actions in the target game based on the current game state data through the target game strategy model; and issue game completion rewards to the agent when the agent meets the preset completion conditions of the target game.

[0060] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0061] An electronic device provided in this application includes a processor and a memory. The memory stores multiple instructions, and the processor loads the instructions to execute the steps in the game strategy model training method provided in this application.

[0062] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps in the game strategy model training method provided in this application.

[0063] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the game strategy model training method provided in embodiments of this application.

[0064] This application provides a game strategy model training method and related equipment. Based on the game state of an agent, a game strategy model of a target game can be used to control the agent's game actions in the target game. The game strategy model determines the actions the agent should perform in different game states. Based on the game state and historical game states, the state information entropy of the agent at the corresponding moment in the current game state is calculated. The historical game states are the states the agent was in before the current game state in the current game. The exploration reward information corresponding to the agent in the current game state is determined based on the state information entropy. Based on the exploration reward information and the game state and actions performed by the agent each time an action is taken, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0065] This application can combine the state information entropy of the agent's game state with the agent's exploration ability during the training process of the game strategy model, and give exploration rewards based on the agent's state information entropy to encourage the agent's curiosity. This can improve the accuracy and reliability of the game action generation of the trained target game strategy model, more accurately identify obstacles in the game process, and thus better control the agent to avoid obstacles in levels with different obstacle placements, thereby improving the agent's obstacle avoidance ability and enhancing the agent's ability to complete levels. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1a This is a schematic diagram of a scenario for the game strategy model training method provided in an embodiment of this application;

[0068] Figure 1b This is a flowchart of the game strategy model training method provided in the embodiments of this application;

[0069] Figure 1c This is an illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0070] Figure 1d This is another illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0071] Figure 1e This is another illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0072] Figure 1f This is another illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0073] Figure 1g This is another illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0074] Figure 1h This is another illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0075] Figure 1i This is another illustrative diagram of the game strategy model training method provided in the embodiments of this application;

[0076] Figure 2 This is another flowchart of the game strategy model training method provided in the embodiments of this application;

[0077] Figure 3 This is a schematic diagram of the structure of the game strategy model training device provided in the embodiments of this application;

[0078] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] This application provides a game strategy model training method and related equipment. The related equipment may include a game strategy model training device, an electronic device, a computer-readable storage medium, and a computer program product. Specifically, the game strategy model training device may be integrated into an electronic device, which may be a terminal or a server, etc.

[0081] It is understood that the game strategy model training method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0082] like Figure 1a As shown, the example illustrates a game strategy model training method where a terminal and a server jointly execute the training. The game strategy model training system provided in this embodiment includes a terminal 10 and a server 11, etc.; the terminal 10 and the server 11 are connected via a network, such as a wired or wireless network connection, etc., wherein the game strategy model training device can be integrated into the server.

[0083] Server 11 can be used to: control the agent's game actions in the target game based on the agent's game state using a game strategy model of the target game, wherein the game strategy model is used to determine the game actions to be performed by the agent in different game states; calculate the state information entropy of the agent at the corresponding moment of the game state based on the game state and historical game states, wherein the historical game states are the states of the agent that appeared before the current game state in the current game; determine the exploration reward information corresponding to the agent in the game state based on the state information entropy; and update the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, to obtain the target game strategy model. Server 11 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0084] Terminal 10 can be used to receive a trained target game strategy model sent by server 11. This target game strategy model is used to control the game actions of the agent during the game of the target game. Terminal 10 can include a mobile phone, vehicle-mounted terminal, aircraft, tablet computer, laptop computer, or personal computer (PC), etc. A client can also be set on terminal 10, which can be an application client or a browser client, etc.

[0085] The steps such as model training performed on the aforementioned server 11 can also be executed by the terminal 10.

[0086] The game strategy model training method provided in this application involves machine learning in the field of artificial intelligence.

[0087] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0088] This embodiment will be described from the perspective of a game strategy model training device, which can be integrated into an electronic device, such as a server or a terminal.

[0089] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0090] like Figure 1b As shown, the specific process of training this game strategy model can be as follows:

[0091] 101. Using the game strategy model of the target game, based on the game state of the agent, control the game actions of the agent in the target game, wherein the game strategy model is used to determine the game actions to be performed by the agent in different game states.

[0092] The target game can be a level-based game, such as a racing game; this embodiment does not limit this. Specifically, each level in the target game has different obstacles, and the agent needs to avoid these obstacles to successfully reach the finish line from the starting point of the level and thus successfully complete the game.

[0093] In this context, an agent refers to a virtual game character controlled by an AI (Artificial Intelligence) model, or a virtual game character controlled by an electronic device. It can engage in battles with other agents or virtual game characters controlled by game players. The AI ​​model here is also known as the game strategy model.

[0094] A game strategy model, also known as a game action decision model, determines the game action an agent should perform based on different game states. It can be a deep learning-based model. Specifically, the game strategy model determines the probability of the agent performing different game actions under different game states. The model can include the probability distribution information of candidate game actions for the agent under different game states. Based on this probability distribution information, the game action randomly selected from the candidate actions is the action the agent will perform. It should be noted that the higher the probability of a candidate action, the greater its chance of being selected; conversely, the lower the probability of a candidate action, the lower its chance of being selected.

[0095] For example, when the action type of the game action is a movement action, in game state 1, the candidate game actions can include moving forward, moving left, moving right, and moving backward. The probability distribution information can be: the selection probability of "moving forward" is 60%, the selection probability of "moving left" is 15%, the selection probability of "moving right" is 15%, and the selection probability of "moving backward" is 10%.

[0096] The game state may include the game environment parameters of the agent, the agent's location in the target game, its own game character attributes, and the current task progress, etc., which are not limited in this embodiment. The action type of the game action may include movement action type, attack action type, defense action type, etc., which are not limited in this embodiment.

[0097] Specifically, the game strategy model can be a model obtained by pre-training the agent's ability to complete the game, and the agent under the control of the game strategy model has the ability to complete the game.

[0098] In current technologies, the mainstream approach to building game AI agents involves designing reasonable level-clearing rewards. This allows the agent to explore and experiment with different actions, learning optimal behavioral strategies to maximize expected rewards and ultimately achieve victory. However, in racing-style UGC (User Generated Content) games, the constant release of new levels makes it impossible to train on all levels using mainstream methods. Training is limited to specific levels, with all levels being applied. Currently, training based solely on level-clearing rewards results in the agent learning the optimal path based on a set reward pattern. This method produces fixed and repetitive paths, leading to poor performance in levels with varying obstacle placements, thus limiting the application of AI agents in racing-style UGC games.

[0099] Current technologies primarily involve adding levels with different obstacles during AI training, requiring approximately 20,000 different obstacle levels. This allows the trained AI to perform well on levels with varying obstacles, making it directly applicable to all levels. While existing technologies guide the agent's obstacle avoidance capabilities by adding a sufficient number of different obstacle levels during training, this requires a large number of different obstacle levels, resulting in high training costs. Furthermore, the selection of diverse obstacle placements in the levels requires significant manual labor costs, making this approach difficult to apply to complex games.

[0100] The game strategy model training method provided in this application can train an agent with obstacle avoidance capabilities, and the training cost is low. In one specific embodiment, this application can first train a game strategy model with level-clearing capabilities, and then train obstacle avoidance capabilities on this basis.

[0101] Optionally, in this embodiment, the step "controlling the agent's game actions in the target game based on the agent's game state using the game strategy model of the target game" may further include:

[0102] Obtain the initial game strategy model of the target game, which is used to determine the game actions to be performed by the agent in different game states;

[0103] Using the initial game strategy model, the agent's game actions in the target game are controlled based on the agent's game state in multiple games of the target game.

[0104] Based on the game state and actions of the agent each time it performs an action that meets the preset clearance conditions, the parameters of the initial game strategy model are updated to obtain a game strategy model. The agent controlled by the game strategy model has the ability to clear the level.

[0105] The initial game strategy model can be an untrained model. Some parameters in the initial game strategy model can also be pre-set. During level completion training, the agent's game state can be input into the initial game strategy model. The initial game strategy model determines the game action the agent should perform, thereby controlling the agent to execute that action, resulting in a new game state. This new game state is then input into the initial game strategy model, which determines a new game action based on the new game state, thus continuing to control the agent to execute the new game action.

[0106] Specifically, the parameters of the initial game strategy model can include the probability distribution information of candidate game actions corresponding to the agent in different game states. When the agent's game state is input into the initial game strategy model, the initial game strategy model can determine each candidate game action corresponding to the game state, and randomly select a candidate game action as the game action to be executed by the agent based on the probability of each candidate game action.

[0107] Specifically, the preset completion condition can be that the agent starts from the starting position of the target game, avoids obstacles, and reaches the finish line to complete the game. In this embodiment, if the agent completes the game in a certain game, the action execution strategy of the agent in that game can be selected. The action execution strategy includes the game state and game action when the agent performs an action each time. This action execution strategy is the target action execution strategy corresponding to the agent that meets the preset completion condition. Based on this target action execution strategy, the initial game strategy model is optimized and updated so that the initial game strategy model learns the ability to complete the game. In a specific scenario, the agent can be controlled to play 10,000 simulated games in the game level through the initial game strategy model. Assuming that the agent reaches the finish line in only 4,000 of these 10,000 games, only in these 4,000 games will the agent's action execution strategy be the target action execution strategy corresponding to the agent that meets the preset completion condition.

[0108] The target action execution strategy includes the game state and game action of the agent each time it performs an action in the current game. Specifically, the target action execution strategy can be parsed to obtain the action data sequence of the agent corresponding to the target action execution strategy. The action data sequence can include various strategy data pairs arranged by execution time. Each strategy data pair includes the game state and game action at the corresponding execution time. For each strategy data pair, the target game state that matches the strategy data pair in the initial game strategy model is determined. Based on the strategy data pair, the probability distribution information of the candidate game actions corresponding to the target game state is updated. Specifically, the probability of candidate game actions that are the same as the game actions in the strategy data pair can be increased, and the probability of candidate game actions that are different from the game actions in the strategy data pair can be decreased. The adjustment range of the probability of candidate game actions can be set according to the actual situation.

[0109] For example, if an agent starts moving from the starting position on the game map of a target game and reaches the finish line after four moves to complete the level, then the agent's target action execution strategy includes the game state and game action (specifically, the movement action) during these four moves. This can be parsed into four policy data pairs; for example, these four policy data pairs are: State A - Move Forward, State B - Move Left, State C - Move Forward, and State D - Move Forward. Based on the policy data pair – State A - Move Forward – the probability of the candidate game action "Move Forward" corresponding to State A in the initial game strategy model can be increased, while the probabilities of other candidate game actions corresponding to State A can be decreased. Based on the strategy data pair—State B - Left Shift—the probability of the candidate game action "Left Shift" corresponding to State B in the initial game strategy model can be increased, while the probability of other candidate game actions corresponding to State B can be decreased. Based on the strategy data pair—State C - Forward—the probability of the candidate game action "Forward" corresponding to State C in the initial game strategy model can be increased, while the probability of other candidate game actions corresponding to State C can be decreased. Based on the strategy data pair—State D - Forward—the probability of the candidate game action "Forward" corresponding to State D in the initial game strategy model can be increased, while the probability of other candidate game actions corresponding to State D can be decreased.

[0110] Specifically, when the agent reaches the end point of the game map, a game completion reward can be issued to the agent. In this embodiment, the action execution strategy of the agent that obtains the game completion reward can be determined as the target action execution strategy, and the initial game strategy model can be updated based on the target action execution strategy.

[0111] Optionally, in this embodiment, the step "updating the parameters of the initial game strategy model based on the game state and game actions of the agent each time it performs an action that meets the preset clearance conditions, to obtain the game strategy model" may include:

[0112] Based on the game result data of the agents in each game, the success rate of the agents in the target game is statistically analyzed.

[0113] When the pass rate does not meet the preset conditions, the parameters of the initial game strategy model are updated based on the game state and game actions of the agent that meets the preset pass conditions each time it performs an action, so as to obtain the updated initial game strategy model.

[0114] Return to the step of controlling the agent's game actions in the target game based on the agent's game state in multiple games of the target game using the initial game strategy model, until the agent's completion rate in the target game meets the preset conditions, thus obtaining the game strategy model.

[0115] Specifically, in each iteration, the agent can play multiple games. For example, in each iteration, training can be performed based on the game results data of 10,000 games played by the agent. In some embodiments, multiple agents can also be set up to play the same game in each iteration, and training can be performed based on the game results data of each agent in that game.

[0116] Specifically, the number of agents that cleared the game in this iteration can be determined based on the game result data of each agent in this iteration. Based on this number and the total number of all agents in this iteration, the clearance rate of agents in this iteration can be obtained.

[0117] The preset condition can be that the success rate of the agents is not less than a preset percentage. The preset percentage can be set according to the actual situation. For example, it can be set to 100%. Only when all agents achieve game completion in this iteration will the training of the initial game strategy model stop and a game strategy model with the ability to complete the game be obtained.

[0118] In this process, if the success rate of agents in the current iteration does not meet the preset conditions, it is necessary to select the target action execution strategies of agents that have achieved game completion in this iteration. Based on the target action execution strategies, the parameters of the initial game strategy model are iteratively updated. Then, the updated initial game strategy model is tested for the agents' ability to complete the game. Specifically, the updated initial game strategy model can be used to control the game actions of multiple agents in the target game to complete the game. After the game ends, the success rate of agents in this round is statistically analyzed. If the statistical results do not meet the preset conditions, the target action execution strategies of agents that have achieved game completion in this round are selected to continue iteratively updating the model's parameters. If the statistical results meet the preset conditions, the updated initial game strategy model passes the test and can be used as a game strategy model with the ability to complete the game.

[0119] In specific scenarios, a certain duration can be set for each game, such as setting the duration of each game to no more than 5 minutes. If an agent has not reached the end position of the game map 5 minutes after the start of the target game, it is considered to have failed to pass the level.

[0120] Specifically, an iteration can be a single game or multiple games, with one or more agents participating in each game. For example, each iteration of the model could involve 10,000 games, with one agent participating in each game. In a specific game scenario, the target game could be a racing level mode, where an agent starts from the starting point on the game map and completes the game by crossing the finish line. Each game could involve one agent, such as... Figure 1c As shown. In each game, the agent's actions on the game map can be controlled by an initial game strategy model. The initial game strategy model determines the agent's actions based on the agent's game state, and then executes the determined actions. If ten thousand games are played, ten thousand movement paths can be obtained through the control of the initial game strategy model. Then, based on these ten thousand movement paths, the agent's success rate is calculated. If the success rate does not meet the preset conditions, movement paths that meet the preset success conditions are selected from these ten thousand movement paths, and the target action execution strategy of the agent corresponding to these selected movement paths is obtained. The parameters of the initial game strategy model are updated according to the target action execution strategy to obtain the updated initial game strategy model. The updated initial game strategy model is then used to play 10,000 games, resulting in 10,000 new movement paths. Based on these 10,000 paths, the success rate of the agent in this iteration is calculated. If the success rate meets a preset condition, training stops, and the updated initial game strategy model is used as a game strategy model with the ability to complete levels. If the success rate still does not meet the preset condition, movement paths that meet the preset condition are selected, and the corresponding target action execution strategies for these paths are obtained. The model is then updated again, and this process is repeated until the success rate meets the preset condition, resulting in a game strategy model with the ability to complete levels. The agent controlled by this game strategy model performs poorly in levels with different obstacles, making it difficult to complete levels. Figure 1d As shown, obstacle avoidance training is also required. To train an agent with obstacle avoidance capabilities, the game strategy model training method provided in this application calculates the state information entropy of the agent's game state to provide exploration rewards, thereby encouraging the agent's curiosity. This enables the trained target game strategy model to better control the agent to avoid obstacles in levels with different obstacle placements, thus improving the agent's obstacle avoidance capabilities.

[0121] Optionally, in this embodiment, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0122] The step "controlling the agent's game actions in the target game based on the agent's game state using the game strategy model of the target game" may include:

[0123] The game state of the agent is input into the game strategy model of the target game to determine at least one candidate game action of the agent;

[0124] The target game action to be performed by the agent is determined from the candidate game actions by using the probability distribution information of the game strategy model.

[0125] Control the intelligent agent to perform target game actions in the target game.

[0126] The game state may include the game environment parameters of the agent, its location, its own game character attributes, and the current task progress, etc. This embodiment does not limit this.

[0127] The probability distribution information includes the probability of performing each candidate game action in a given game state. For example, in game state 1, candidate game actions may include moving forward, moving left, moving right, and moving backward, and their probability distribution information may be: the probability of selecting "moving forward" is 60%, the probability of selecting "moving left" is 15%, the probability of selecting "moving right" is 15%, and the probability of selecting "moving backward" is 10%.

[0128] The higher the probability of a candidate game action, the greater the chance of it being selected; conversely, the lower the probability of a candidate game action, the lower the chance of it being selected.

[0129] 102. Based on the game state and the historical game state, calculate the state information entropy of the agent at the time corresponding to the game state, wherein the historical game state is the state in which the agent appeared before the game state in the current game.

[0130] Optionally, in this embodiment, the step "calculating the state information entropy of the agent at the corresponding moment of the game state based on the game state and the historical game states" may include:

[0131] Determine the historical state sequence corresponding to the game state, wherein the historical state sequence includes at least one historical game state corresponding to the game state and the game state;

[0132] Each state in the historical state sequence is processed by state encoding to obtain the state encoding information of each state in the historical state sequence;

[0133] The state coding information of each state is classified and processed to determine the proportion of occurrence of each state under each preset state type;

[0134] Based on the occurrence percentage information of each preset state type, the type information entropy of the historical state sequence is calculated to obtain the state information entropy of the agent at the corresponding moment of the game state.

[0135] In some embodiments, the historical game states corresponding to a game state can specifically be all game states that the agent has experienced before the current game state in the current game. For example, if the agent has performed 14 game actions in the current game of the target game and is currently in the 15th game state, then the previous 14 game states are the historical game states of the 15th game state, and the historical state sequence can include the 14 historical game states and the current 15th game state. In other embodiments, the historical game states corresponding to a game state can specifically be the n game states that the agent experienced before the current game state in the current game. n can be set according to the actual situation, such as 10. For example, if the agent has performed 14 game actions in the current game of the target game and is currently in the 15th game state, then the historical game states of the 15th game state can be the 10 game states that occurred before the 15th game state, that is, game states 5 to 14 are the historical game states of the 15th game state, and the historical state sequence can include these 10 historical game states and the current 15th game state.

[0136] In the historical state sequence, each game state can be sorted according to the time of its occurrence.

[0137] Specifically, state coding can be divided into two coding methods: fixed coding and learnable coding, which are unified as follows: Figure 1e express.

[0138] Fixed encoding, also known as hard encoding, is a relatively simple and interpretable method. The encoding functions in fixed encoding can be directly set by domain experts based on experience. For example, in racing games, to enable an agent to learn obstacle avoidance capabilities, state encoding can be based on the agent's corresponding coordinate positions. Figure 1f It identifies the spatial coordinates (x, y, z) of the agent from the game scene and then determines the corresponding state coding information based on these spatial coordinates.

[0139] Specifically, in addition to coordinate encoding, fixed encoding can also use other encoding methods with discriminative capabilities. For example, state encoding can be based on whether the agent is blocked by obstacles. If the current game state is blocked by obstacles, it is encoded as 1; if the current game state is not blocked by obstacles, it is encoded as 0. With this encoding method, if the state encoding of the agent's game state is always 1, it means that the agent is blocked by obstacles and cannot move forward. At this time, the state information entropy corresponding to the game state is also low, resulting in low exploration reward information. Thus, the action execution strategy of the agent in this process is abandoned.

[0140] The learnable encoding function does not require prior knowledge from domain experts. Instead, it defines the encoding function as a parameterized function, which can be a neural network or other types. Specifically, it can define a loss function based on the requirements of the encoding space and learn the encoding function on a sufficient dataset. Figure 1g As shown, the historical states determined from the game scene are used as input to the learned parameterized encoding function, which then outputs the state encoding information corresponding to that historical state. The advantage of this method is that it can learn discriminative encodings without requiring prior knowledge.

[0141] In this process, state encoding is used to assign a state code to each state in the historical state sequence. The state codes for different states may be the same or different. Specifically, in some embodiments, multiple preset state codes can be pre-set, such as five preset state codes. Through state encoding, the state code corresponding to each state in the historical state sequence can be determined from these five preset state codes. Specifically, the state code corresponding to each state can be determined by the matching degree between each state and each preset state code. For example, the game map of the target game includes multiple areas, and each area can correspond to a preset state code. For instance, the preset state code for area A can be 1, and the preset state code for area B can be 2. Game states in the same area have the same state code. A preset state code can be considered as a state identifier or state label corresponding to a state type.

[0142] In some embodiments, the state coding information of each state is classified and processed. Specifically, based on the state coding information of each state, the preset state type to which each state belongs is determined. Then, the number of states under each preset state type is counted. Finally, based on the total number of states in the historical state sequence and the number of states under each preset state type, the occurrence ratio information of states under each preset state type is determined. The preset state type to which each state belongs can be determined based on the similarity between the state coding information of each state and the preset state coding information corresponding to each preset state type.

[0143] For example, if the state codes of each state in the historical state sequence are 1, 3, 4, 3, 1, 2, 5, 1, 5, 3, and there are five preset state types, with their corresponding state identifiers (i.e., preset state codes) being 1, 2, 3, 4, and 5, then we can determine that there are 3 states belonging to preset state type 1, 1 state belonging to preset state type 2, 3 states belonging to preset state type 3, 1 state belonging to preset state type 4, and 2 states belonging to preset state type 5 in the historical state sequence. Since the total number of states in the historical state sequence is 10, the percentage of states appearing under preset state type 1 is 3 / 10, under preset state type 2 is 1 / 10, under preset state type 3 is 3 / 10, under preset state type 4 is 1 / 10, and under preset state type 5 is 2 / 10.

[0144] After obtaining the occurrence percentage information of each preset state type, the type information entropy of the historical state sequence of the game state can be calculated based on the occurrence percentage information to obtain the state information entropy of the agent at the corresponding moment of the game state. As shown in equation (1), the calculation method of state entropy information is as follows:

[0145]

[0146] Where n represents the number of preset state types, p i H(X) represents the percentage of states belonging to the preset state type i in the historical state sequence corresponding to a certain game state, and H(X) represents the state information entropy of the agent at the corresponding moment of the game state.

[0147] Optionally, in this embodiment, the action type of the intelligent agent's game action includes movement action type;

[0148] The step "perform state encoding processing on each state in the historical state sequence to obtain state encoding information for each state in the historical state sequence" may include:

[0149] Obtain the position information of the agent in the target game under each state of the historical state sequence;

[0150] Based on the location information, determine the state encoding information of each state in the historical state sequence.

[0151] In this embodiment, the position information in each state can be directly used as the state encoding information. This is because the agent moves in the game map of the target game, so the position of the agent is usually constantly changing. The change in position also reflects the agent's exploration ability and indicates that the agent is not blocked by obstacles. Therefore, using the position information as the state encoding information to calculate the state information entropy and determining the exploration reward information based on the state information entropy can encourage the agent's curiosity. If the agent's position does not change, it means that the agent is blocked by obstacles, and its state information entropy is low.

[0152] Optionally, in this embodiment, the step "performing state encoding processing on each state in the historical state sequence to obtain state encoding information for each state in the historical state sequence" may include:

[0153] The state coding model is used to perform state coding processing on each state in the historical state sequence to obtain the state coding information of each state in the historical state sequence.

[0154] Before performing state encoding processing on each state in the historical state sequence using a state encoding model to obtain the state encoding information of each state in the historical state sequence, the method further includes:

[0155] Acquire training data, which includes at least one sample game state of the agent in the target game and the expected state encoding information corresponding to the sample game state;

[0156] The sample game state is processed by a state coding model to obtain the sample state coding information of the sample game state.

[0157] Based on the sample state encoding information and the desired state encoding information, the parameters of the state encoding model are adjusted to obtain the trained state encoding model.

[0158] The state encoding model can be a neural network model or a tree model; this embodiment does not impose any specific limitations on it. Through the state encoding model, state feature information of the sample game state can be extracted. Then, based on the matching degree between the state feature information and each preset state type, the sample state encoding information is determined based on the preset state encoding information corresponding to the matched preset state type.

[0159] In the training process of the state coding model, the loss information between the sample state coding information and the desired state coding information can be calculated first. Based on this loss information, the parameters of the state coding model are adjusted using the backpropagation algorithm. Based on this loss information, the parameters of the state coding model are optimized so that the sample state coding information approaches the desired state coding information, thus obtaining a trained state coding model. Specifically, the loss information can be made to be less than a preset loss value, which can be set according to the actual situation.

[0160] Optionally, in this embodiment, the game strategy model training method may further include:

[0161] Based on the information regarding the exploration capabilities required by the intelligent agent, the preset encoding space of the game state of the intelligent agent is divided to obtain multiple state encoding spaces, each state encoding space corresponding to a state encoding information.

[0162] Based on the state encoding space in which the sample game state is located, determine the expected state encoding information corresponding to the sample game state.

[0163] In this scenario, if the agent's exploratory capabilities are highly demanding, the preset coding space can be divided more finely, increasing the number of state coding spaces. Different state coding spaces contain different state coding information, while game states within the same state coding space share the same state coding information. This setup increases state information entropy, thereby better encouraging the agent's curiosity. Conversely, if the agent's exploratory capabilities are relatively low, the number of state coding spaces can be reduced. This embodiment, by defining the state coding space, controls the scope of curiosity, thus better controlling the agent's performance.

[0164] In a specific scenario, the preset encoding space of the game state can be regarded as the range of values ​​of the game state. If the game state is location information, then the preset encoding space can be the game map of the target game, and each state encoding space can correspond to a small area.

[0165] In other embodiments, the state coding model can be trained without using the desired state coding information. Instead, it can be trained using positive and negative samples of the game state.

[0166] Optionally, in some embodiments, before the step "performing state encoding processing on each state in the historical state sequence using a state encoding model to obtain state encoding information for each state in the historical state sequence", the following may be included:

[0167] Acquire training data, which includes at least one sample game state of the agent in the target game, as well as positive and negative sample game states of the sample game states.

[0168] Using a state coding model, the sample game state, the positive sample game state, and the negative sample game state are respectively processed by state coding to obtain the first state coding information of the sample game state, the second state coding information of the positive sample game state, and the third state coding information of the negative sample game state.

[0169] Based on the similarity between the first state encoding information and the second state encoding information, and the similarity between the first state encoding information and the third state encoding information, the parameters of the state encoding model are adjusted to obtain the trained state encoding model.

[0170] Specifically, in some embodiments, game states that are geographically close to the sample game state can be defined as positive sample game states. "Genuinely close" here specifically means that the geographical distance is less than a first preset distance. Game states that are geographically closer to the sample game state than a second preset distance can be defined as negative sample game states. The first preset distance is less than the second preset distance. In other embodiments, game states that are in the same game area as the sample game state can be defined as positive sample game states, and game states that are in different game areas can be defined as negative sample game states.

[0171] In this embodiment, the similarity between the first and second state encoding information is the positive sample pair similarity, and the similarity between the first and third state encoding information is the negative sample pair similarity. Based on the positive and negative sample pair similarities, the backpropagation algorithm is used to adjust the parameters of the state encoding model. The parameters of the state encoding model are optimized based on these similarities, so that the positive sample pair similarity approaches the expected similarity of positive sample pairs, and the negative sample pair similarity approaches the expected similarity of negative sample pairs, thus obtaining a well-trained state encoding model. Specifically, the calculated positive sample pair similarity can be higher than the expected similarity of positive sample pairs, and the negative sample pair similarity can be lower than the expected similarity of negative sample pairs. The expected similarity of positive and negative sample pairs can be set according to actual conditions.

[0172] 103. Based on the state information entropy, determine the exploration reward information corresponding to the agent in the game state.

[0173] There are several ways to determine reward information. Specifically, rewards can be divided into hard rewards and soft rewards. Hard rewards are given based on whether the entropy value meets a preset threshold, while soft rewards are given based on the entropy value and a preset increasing function, with the function value of the entropy being the reward.

[0174] Optionally, in this embodiment, the step "determining the exploration reward information corresponding to the agent in the game state based on the state information entropy" may include:

[0175] When the state information entropy is greater than a preset threshold, the exploration reward information corresponding to the agent in the game state is determined as the first exploration reward information;

[0176] When the state information entropy is not greater than a preset threshold, the exploration reward information corresponding to the agent in the game state is determined to be the second exploration reward information, and the second exploration reward information is less than the first exploration reward information.

[0177] In this embodiment, exploration reward information is determined through hard rewards. Hard rewards are given based on whether the entropy value meets a preset threshold, which can be set according to actual conditions. Hard rewards are simple to calculate, highly interpretable, and the guided agent will only show strong curiosity when the entropy value meets the threshold condition. Figure 1h As shown, this illustrates curiosity guided by hard rewards. When the agent encounters an obstacle, its game state changes relatively significantly, resulting in a higher state information entropy. Under this hard reward approach, a larger first exploration reward can be given. Conversely, when the agent does not encounter an obstacle, its game state changes relatively little, and its state information entropy may not exceed a preset threshold. When the state information entropy does not exceed the preset threshold, a smaller second exploration reward is given. Thus, only when the agent encounters an obstacle will the exploration reward change significantly, thereby stimulating the agent's curiosity.

[0178] It is understandable that the second exploration reward information here may not necessarily be a reward; it could be a punishment, or it could be that no reward or punishment is given at all.

[0179] Specifically, the hard reward is calculated as shown in equation (2):

[0180]

[0181] Where H(X) represents the state information entropy, θ represents the preset threshold, and reward represents the exploration reward information. When H(X)>θ, the exploration reward information is determined as the first exploration reward information 1; when H(X)≤θ, the exploration reward information is determined as the second exploration reward information 0.

[0182] It is understandable that, in addition to calculation using a single threshold, hard rewards can also be calculated using multiple thresholds, and this embodiment does not impose any restrictions on this.

[0183] Optionally, in this embodiment, the step "determining the exploration reward information corresponding to the agent in the game state based on the state information entropy" may include:

[0184] Get the preset increment function;

[0185] The state information entropy is processed by the preset incrementing function to obtain the exploration reward information corresponding to the game state of the intelligent agent.

[0186] The preset increasing function can be any increasing function. This embodiment determines the exploration reward information through soft rewards, which are based on the entropy value and the preset increasing function, using the entropy function value as the reward. The advantage of this calculation method is that there is a reward value at every moment, guiding the agent to have strong curiosity, and causing it to move towards regions with higher entropy values ​​at any given time. Figure 1i This demonstrates how soft rewards guide curiosity. Under the soft reward calculation method, different state information entropy corresponds to different exploration reward information. The exploration reward information is given according to the state information entropy at different times. Since the exploration reward information given at each time is basically different, the agent's game actions can be rewarded or punished according to the differences in exploration reward information at each time. This can better guide the curiosity of the agent's game actions at each time and better stimulate the agent's curiosity.

[0187] Specifically, the soft reward is calculated as shown in equation (3):

[0188]

[0189] Here, H(X) represents the state information entropy, which is non-negative and can be 0 or a positive number, and reward represents the exploration reward information.

[0190] The game strategy model training method of this application can be based on exploration reward information for model training. In this way, during the training process, in order to obtain more rewards, the agent will explore areas with higher entropy values, thereby enhancing the agent's curiosity and improving the agent's obstacle avoidance ability.

[0191] 104. Based on the exploration reward information and the game state and game actions of the agent each time it performs an action, update the parameters of the game strategy model to obtain the target game strategy model.

[0192] The target game strategy model is used to control the agent's game actions during the target game. By obtaining the final target game strategy model, the agent's obstacle avoidance ability can be improved, enabling the agent to have a stronger ability to complete levels with different obstacle placements.

[0193] In this embodiment, the curiosity reward can be calculated and given to the agent in real time, or it can be given to the agent in other ways, such as giving the agent a reward at the end of the game.

[0194] Optionally, in this embodiment, the step "updating the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, to obtain the target game strategy model" may include:

[0195] Determine the associated game states corresponding to each game state of the agent in this game;

[0196] For each game state, based on the exploration reward information corresponding to the game state and the exploration reward information corresponding to the associated game state, the action reward and penalty information corresponding to the game state is determined.

[0197] Based on the game state and game action of the agent each time it performs an action, as well as the reward and penalty information of the action corresponding to each game state of the agent in this game, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0198] Specifically, the associated game state of a certain game state can be the state corresponding to the execution of a certain game action based on that game state. The associated game state is the next game state of that game state in the current game.

[0199] Specifically, the reward / penalty information for a particular game state refers to the reward or penalty information for game actions performed based on that game state. This information can be determined by the difference between the exploration reward information of the associated game states and the exploration reward information of the current game state. If the exploration reward information of the next game state is smaller than that of the current game state, it indicates that the game action performed based on that state has reduced the state information entropy, meaning the exploration aspect of that action is lower, and therefore a penalty is imposed on that action.

[0200] Optionally, in this embodiment, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0201] The step "updating the parameters of the game strategy model based on the game state and game action of the agent each time it performs an action, and the action reward and penalty information corresponding to each game state of the agent in this game, to obtain the target game strategy model" may include:

[0202] Construct the action data sequence of the agent, the action data sequence including each strategy data pair arranged by execution time in the current game, the strategy data pair including the game state and game action at the corresponding execution time;

[0203] For each strategy data pair, determine the target game state in the game strategy model that matches the strategy data pair;

[0204] Based on the action reward and punishment information corresponding to the game state in the strategy data pair, the probability distribution information of the candidate game actions corresponding to the target game state is updated to obtain the target game strategy model.

[0205] The parameters of the game strategy model can include the probability distribution information of candidate game actions corresponding to the agent in different game states. When the agent's game state is input into the game strategy model, the game strategy model can determine each candidate game action corresponding to the game state, and randomly select a candidate game action as the game action to be executed by the agent based on the probability of each candidate game action.

[0206] The target game state is the game state in the game strategy model that is the same as the game state in the strategy data pair.

[0207] Specifically, the probabilities of candidate game actions corresponding to the same game state as the game state in the strategy data pair can be adjusted. If the reward / penalty information for a game action in the strategy data pair is a penalty, then for candidate game actions corresponding to the target game state, the probability of candidate game actions that are the same as those in the strategy data pair will be decreased, while the probability of candidate game actions that are different from those in the strategy data pair will be increased. If the reward / penalty information for a game action in the strategy data pair is a reward, then for candidate game actions corresponding to the target game state, the probability of candidate game actions that are the same as those in the strategy data pair will be increased, while the probability of candidate game actions that are different from those in the strategy data pair will be decreased. The magnitude of the probability adjustment for candidate game actions can be set according to the magnitude of the reward / penalty information.

[0208] For example, the action execution strategy of an agent includes the game state and game action when four actions are executed. Specifically, the action execution strategy can be parsed into four policy data pairs; for example, these four policy data pairs are: state A - move forward, state B - move left, state C - move forward, and state D - move forward.

[0209] For a strategy data pair—State A-Forward—if the reward / penalty information for the game action "Forward" is a reward, then the probability of the candidate game action "Forward" corresponding to State A in the game strategy model can be increased, while the probability of other candidate game actions corresponding to State A can be decreased. If the reward / penalty information for the game action "Forward" is a penalty, then the probability of the candidate game action "Forward" corresponding to State A in the game strategy model can be decreased, while the probability of other candidate game actions corresponding to State A can be increased.

[0210] Based on the strategy data pair – State B – Left Shift: If the reward / penalty information for the game action “Left Shift” is a reward, then the probability of the candidate game action “Left Shift” corresponding to State B in the game strategy model can be increased, while the probability of other candidate game actions corresponding to State B can be decreased; if the reward / penalty information for the game action “Left Shift” is a penalty, then the probability of the candidate game action “Left Shift” corresponding to State B in the game strategy model can be decreased, while the probability of other candidate game actions corresponding to State B can be increased.

[0211] Based on the strategy data pair – State C – Moving Forward: If the reward / penalty information for the game action "Move Forward" is a reward, then the probability of the candidate game action "Move Forward" corresponding to State C in the game strategy model can be increased, while the probability of other candidate game actions corresponding to State C can be decreased; if the reward / penalty information for the game action "Move Forward" is a penalty, then the probability of the candidate game action "Move Forward" corresponding to State C in the game strategy model can be decreased, while the probability of other candidate game actions corresponding to State C can be increased.

[0212] Based on the strategy data pair – State D – Moving Forward: If the reward / penalty information for the game action "Move Forward" is a reward, then the probability of the candidate game action "Move Forward" corresponding to State D in the game strategy model can be increased, while the probability of other candidate game actions corresponding to State D can be decreased; if the reward / penalty information for the game action "Move Forward" is a penalty, then the probability of the candidate game action "Move Forward" corresponding to State D in the game strategy model can be decreased, while the probability of other candidate game actions corresponding to State D can be increased.

[0213] This embodiment can encourage the agent to perform game actions with high entropy values ​​through action reward and punishment information, thereby encouraging the agent's curiosity, improving the agent's exploration ability, and thus enhancing the agent's obstacle avoidance ability.

[0214] Optionally, in this embodiment, the step "updating the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, to obtain the target game strategy model" may include:

[0215] Based on the exploration reward information corresponding to the current game state of the agent, and the exploration reward information of the game state corresponding to the current game state after the agent performs the current game action, the action reward and punishment information corresponding to the current game state is determined;

[0216] The game strategy model is updated with parameters based on the action reward and punishment information of the current game state, the current game state, and the game actions performed by the agent in the current game state.

[0217] The game state after the agent performs the game action is taken as the new current game state;

[0218] Using the updated game strategy model, based on the agent's new current game state, the game actions of the agent in the target game are controlled, and the step of calculating the state information entropy of the agent at the corresponding moment of the game state based on the game state and historical game states is returned to execute, so as to obtain the target game strategy model.

[0219] The reward and penalty information for the current game state can be determined by the difference between the exploration reward information of the game state corresponding to the current game state after the agent performs the current game action and the exploration reward information corresponding to the current game state.

[0220] Specifically, in this embodiment, obstacle avoidance training is performed by calculating rewards in real time and providing them to the agent. The agent's current game state is input into the game strategy model, which determines the game action the agent needs to perform. This controls the agent to execute the action in the target game, thus obtaining a new game state for the agent. Based on the current game state and the agent's new game state, the action reward and penalty information corresponding to the current game state can be calculated in real time, and the parameters of the game strategy model are updated accordingly. Then, the agent's new game state is input into the updated game strategy model to determine the new game action to be performed. Based on the new game action, the agent's latest game state is obtained, and a new round of action reward and penalty information calculation is performed to update the parameters of the game strategy model.

[0221] Optionally, in this embodiment, the step "updating the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, to obtain the target game strategy model" may include:

[0222] Based on the exploration reward information and the game state and game actions of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the updated game strategy model.

[0223] Based on the state information entropy corresponding to each game state of the agent in the target game, the updated game strategy model is processed to evaluate the agent's obstacle avoidance ability.

[0224] When the updated game strategy model passes the evaluation, it is determined as the target game strategy model.

[0225] In some embodiments, the obstacle avoidance ability of the intelligent agent can be evaluated based on the state information entropy of each game state of the intelligent agent. If the state information entropy of the game state of the intelligent agent is low, it means that the intelligent agent may be stuck in one place and has poor obstacle avoidance ability.

[0226] In other embodiments, the obstacle avoidance ability of the agent can also be evaluated directly by detecting the agent's performance in the target game. For example, if there is an obstacle in the middle of the game map, and the agent moves around the obstacle to either side, its obstacle avoidance ability can be considered relatively strong. If the agent remains stuck at the obstacle, it indicates that its obstacle avoidance ability is relatively weak.

[0227] Specifically, in some embodiments, when the evaluation result shows that the state information entropy corresponding to each game state is greater than a preset value, or the number of game states with state information entropy greater than the preset value meets a preset quantity condition, the updated game strategy model can be considered to have passed the evaluation. The preset value and preset quantity condition can be set according to actual circumstances.

[0228] In a specific scenario, an agent with the ability to complete levels can be trained on obstacle avoidance in 10,000 games. The evaluation of the agent's obstacle avoidance ability can be determined based on its performance in these 10,000 games. If the number of games in which the agent's performance passes the obstacle avoidance ability evaluation is greater than a preset number, then the updated game strategy model can be determined as the target game strategy model. The preset number of games can be 9,000, but this embodiment does not impose this limitation.

[0229] In this embodiment, training of the game strategy model is stopped only when the agent's obstacle avoidance ability passes the evaluation in the current iteration, resulting in a target game strategy model with obstacle avoidance capabilities. Specifically, if the evaluation fails, the process returns to the steps of controlling the agent's game actions in the target game based on the agent's game state, after the game strategy model has passed the evaluation, until the updated game strategy model passes the obstacle avoidance ability evaluation.

[0230] After obtaining a target game strategy model with obstacle avoidance capabilities, the model can be put into the game application for control of the intelligent agent.

[0231] Optionally, in this embodiment, after the step "updating the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, to obtain the target game strategy model", the game strategy model training method may further include:

[0232] Obtain the current game state data of the agent in the target game;

[0233] The current game state data of the agent is input into the target game strategy model so that the target game strategy model can control the agent's game actions in the target game based on the current game state data.

[0234] When the agent meets the preset completion conditions of the target game, a game completion reward is issued to the agent.

[0235] It should be noted that in the application of the target game strategy model, it is not necessary to calculate exploration reward information, etc.

[0236] Specifically, the game strategy model training method of this application can be divided into three main modules: an agent training module, a state encoding module, and a curiosity reward module. First, the agent training module enables the agent to acquire the ability to complete levels. Based on the state encoding module, the agent's historical game states are encoded and used as features input to the agent. The curiosity reward module calculates the entropy value based on the agent's current game state encoding and historical game state encoding, and provides rewards based on the entropy value, encouraging the agent to move towards states with higher entropy values, i.e., learning curiosity. This allows the agent to navigate obstacles in different levels with varying obstacle placements, based on its historical game states, thus enabling it to achieve obstacle avoidance while maintaining its ability to complete levels.

[0237] The agent training module can design corresponding level-clearing rewards based on the actual game scenario. Specifically, when the agent starts from the starting position on the game map and reaches the finish line, the game is considered cleared, and a level-clearing reward is awarded. (See reference...) Figure 1cBased on this reward, an agent is trained using a reinforcement learning algorithm. Specifically, the agent can play a corresponding number of game levels, such as 10,000, and then the agent's completion rate in each level is calculated. If the completion rate is below 100%, the target action execution policy of the agent that received the completion reward in this iteration is selected, and the model is updated using a reinforcement learning algorithm based on this target action execution policy. Then, the updated model is used to continue playing the corresponding number of game levels to train its completion ability. This process is repeated until the agent's completion rate reaches 100%, resulting in an agent with completion ability. While this method yields an agent with completion ability, relying solely on completion rewards, the agent obtained at this stage performs poorly in levels with different obstacles and struggles to complete them. Figure 1d As shown, obstacle avoidance ability also needs to be trained through a state coding module and a curiosity reward module.

[0238] The reinforcement learning (RL) algorithm can be PPO (Proximal Policy Optimization), Deep Deterministic Policy Gradient (DDPG), or Asynchronous Advantage Actor-Critic (A3C), etc., and this embodiment does not limit it.

[0239] The state encoding module primarily encodes the agent's game state and uses this as a feature input to the agent. Depending on the specific scenario and requirements, it can be divided into fixed encoding and learnable encoding. In a concrete scenario, the agent's historical states can be identified through the game scene visuals. The game scene visuals include the scene information in which the agent is located. Based on this scene information, the agent's location, game environment parameters, game character attributes, and current task progress—all information related to the game state—can be determined. (Reference) Figure 1e The current time is t+1. We can obtain game scene screen 1 corresponding to time t-2, game scene screen 2 corresponding to time t-1, and game scene screen 3 corresponding to time t. From game scene screen 1, we can identify the historical game state at time t-2, denoted as S. t-2 From game scene screen 2, the historical game state at time t-1 can be identified and denoted as S. t-1 From game scene screen 3, the historical game state at time t can be identified and denoted as S. t Then, S respectively t-2 S t-1 S tThe input is fed into the status coding module for status coding, and the corresponding historical status codes of the three can be obtained.

[0240] In the fixed encoding, the encoding function can be directly set by domain experts based on experience. For example, in a racing scenario, to enable the agent to learn obstacle avoidance capabilities, the agent's historical states can be encoded into corresponding coordinate positions, such as... Figure 1f The system identifies the agent's spatial coordinates (x, y, z) from the game scene and then uses these coordinates to determine the corresponding state encoding information. The advantages of this encoding method are its simplicity and high interpretability.

[0241] In this learnable encoding method, the encoding function does not require prior knowledge from domain experts. Instead, the encoding function is defined as a parameterized encoding function, which can be a neural network or other types. A loss function is defined based on the requirements of the encoding space, and this parameterized encoding function is learned on a sufficient dataset. For example... Figure 1g As shown, the historical states determined from the game scene are used as input to the learned parameterized encoding function, which then outputs the state encoding information corresponding to that historical state. The advantage of this method is that it can learn discriminative encodings without requiring prior knowledge.

[0242] For the curiosity reward module, this module is based on the state encoding module, which calculates the entropy value by encoding the agent's current state and historical state, as shown in the above formula (1); then, exploration reward information is given according to the size of the entropy value, and the reward method can be divided into hard reward and soft reward.

[0243] Hard reward refers to giving 1 if the entropy value meets the preset threshold, and giving 0 otherwise. Soft reward refers to using the function value of entropy as the reward based on the entropy value and the increasing function, as shown in the above formula (3). The advantage of this calculation method is that there will be a reward value at every moment, and the guided agent has a strong curiosity and will move to the region with a larger entropy value at any time.

[0244] Specifically, after defining the reward method, during the agent training process, each time the agent plays a game, the agent's curiosity reward is calculated in real time and given to the agent to guide the agent to learn curiosity.

[0245] In one specific embodiment, the training process of the game strategy model training method provided in this application can be as follows:

[0246] Step 1: Based on the game scenario and the intelligent agent training module, design a level completion reward to facilitate the learning of the level completion path.

[0247] Step 2: Based on the reinforcement learning-based PPO algorithm, the agent is trained by playing 10,000 simulated games in game levels using a game strategy model.

[0248] Step 3: Calculate the agent's level completion rate. If the completion rate is less than 100%, select the action execution strategies of the agents that have completed the levels. Adjust the model parameters based on these action execution strategies and return to Step 2 to continue training. If the completion rate reaches 100%, an agent with the ability to complete the levels can be obtained, and the next learning step can be started.

[0249] Step 4: Based on the scenario and requirements, define a state encoding function to encode the game state, and learn the encoding function parameters according to the task objectives.

[0250] Step 5: Based on the scenario and needs, choose the implementation method of curiosity reward.

[0251] Step 6: Continue training the agent with the ability to complete the level for another 10,000 rounds. During the training process, based on the state encoding function, calculate the curiosity reward value in real time in each round of each training session, and reward the agent to guide its curiosity.

[0252] Step 7: Evaluate the obstacle avoidance ability of the agent. If the obstacle avoidance ability meets the requirements, end the training to obtain an agent with obstacle avoidance ability; otherwise, return to step 6 for further training.

[0253] It should be noted that the reward for completing the level can be given after reaching the finish line, or other rewards can be given to encourage completion, such as giving a reward for moving a certain distance towards the finish line.

[0254] The technical solution of this application is mainly based on a curiosity-guided obstacle avoidance learning method for intelligent agents. By defining a state encoding function for an intelligent agent with the ability to pass levels, the current state and historical state of the agent are encoded. Based on the encoded vector, the entropy value of the state is calculated. The curiosity module calculates the curiosity reward of the intelligent agent, which guides the curiosity of the intelligent agent and increases the obstacle avoidance ability of the intelligent agent. Ultimately, the intelligent agent can avoid obstacles in levels with changing obstacles, thus having a strong ability to pass levels.

[0255] Compared to traditional methods that require numerous obstacle courses to train intelligent agents' obstacle avoidance capabilities, this application's curiosity-guided obstacle avoidance learning method can train an intelligent agent with obstacle avoidance capabilities without adding extra training courses, thus saving training costs. Furthermore, this application ensures that the resulting curious intelligent agent also possesses obstacle-avoidance capabilities through training and adjustments to an agent already capable of overcoming obstacles.

[0256] Furthermore, this application can control the strength of an agent's curiosity through soft or hard rewards, thereby better controlling the agent's performance. Specifically, with soft rewards, a reward value is given at each moment, which stimulates the agent to have stronger curiosity.

[0257] As can be seen from the above, this embodiment can control the game actions of the agent in the target game based on the game state of the agent through the game strategy model of the target game. The game strategy model is used to determine the game actions to be performed by the agent in different game states. Based on the game state and the historical game state, the state information entropy of the agent at the corresponding moment of the game state is calculated. The historical game state is the state of the agent before the current game state. According to the state information entropy, the exploration reward information corresponding to the agent in the game state is determined. Based on the exploration reward information and the game state and game action of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0258] This application can combine the state information entropy of the agent's game state with the agent's exploration ability during the training process of the game strategy model, and give exploration rewards based on the agent's state information entropy to encourage the agent's curiosity. This can improve the accuracy and reliability of the game action generation of the trained target game strategy model, more accurately identify obstacles in the game process, and thus better control the agent to avoid obstacles in levels with different obstacle placements, thereby improving the agent's obstacle avoidance ability and enhancing the agent's ability to complete levels.

[0259] Based on the method described in the preceding embodiments, the following will provide a more detailed explanation by taking the specific integration of the game strategy model training device into the server as an example.

[0260] This application provides a method for training a game strategy model, such as... Figure 2 As shown, the specific process of training this game strategy model can be as follows:

[0261] 201. The server obtains the game strategy model of the target game, which is used to determine the game actions to be performed by the agent in different game states.

[0262] The game strategy model can determine the probability of an agent performing different game actions under different game states. The game strategy model can include the probability distribution information of candidate game actions corresponding to the agent under different game states. Based on the probability distribution information, the game action randomly selected from the candidate game actions is the game action to be performed by the agent. It should be noted that the higher the probability of the candidate game action, the higher the probability of it being selected, and vice versa.

[0263] Specifically, the game strategy model can be a model obtained by pre-training the agent's ability to complete the game, and the agent under the control of the game strategy model has the ability to complete the game.

[0264] Optionally, in this embodiment, the step "obtaining the game strategy model of the target game" may include:

[0265] Obtain the initial game strategy model of the target game, which is used to determine the game actions to be performed by the agent in different game states;

[0266] Using the initial game strategy model, in multiple games of the target game, the game actions of the agent are controlled based on the game state of the agent in the target game in order to pass the target game.

[0267] Obtain the target action execution strategy corresponding to the intelligent agent that meets the preset level-clearing conditions. The target action execution strategy includes the game state and game action of the intelligent agent each time it performs an action.

[0268] Based on the target action execution strategy, the parameters of the initial game strategy model are updated to obtain a game strategy model. The agent controlled by the game strategy model has the ability to complete the level.

[0269] The initial game strategy model can be an untrained model. Some parameters in the initial game strategy model can also be pre-set.

[0270] Optionally, in this embodiment, the step "updating the parameters of the initial game strategy model based on the target action execution strategy to obtain the game strategy model" may include:

[0271] Based on the game result data of the agents in each game, the success rate of the agents in the target game is statistically analyzed.

[0272] When the success rate does not meet the preset conditions, the parameters of the initial game strategy model are updated based on the target action execution strategy to obtain the updated initial game strategy model.

[0273] Return to the step of controlling the agent's game actions in the target game based on the agent's game state in multiple games of the target game using the initial game strategy model, until the agent's completion rate in the target game meets the preset conditions, thus obtaining the game strategy model.

[0274] Specifically, the number of agents that cleared the game in this iteration can be determined based on the game result data of each agent in this iteration. Based on this number and the total number of all agents in this iteration, the clearance rate of agents in this iteration can be obtained.

[0275] The preset condition can be that the success rate of the agents is not less than a preset percentage. The preset percentage can be set according to the actual situation. For example, it can be set to 100%. Only when all agents achieve game completion in this iteration will the training of the initial game strategy model stop and a game strategy model with the ability to complete the game be obtained.

[0276] 202. The server, through the game strategy model, controls the game actions of the agent in the target game based on the agent's game state, so as to complete the target game.

[0277] Optionally, in this embodiment, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0278] The step "controlling the agent's game actions in the target game based on the agent's game state using the game strategy model" may include:

[0279] The game state of the agent is input into the game strategy model to determine at least one candidate game action of the agent;

[0280] The target game action to be performed by the agent is determined from the candidate game actions by using the probability distribution information of the game strategy model.

[0281] Control the intelligent agent to perform target game actions in the target game.

[0282] The game state may include the game environment parameters of the agent, its location, its own game character attributes, and the current task progress, etc. This embodiment does not limit this.

[0283] 203. The server determines the historical state sequence corresponding to the game state. The historical state sequence includes at least one historical game state corresponding to the game state and the game state. The historical game state is the state in which the agent appeared before the game state in the current game.

[0284] In some embodiments, the historical game states corresponding to the game state can specifically be all the game states that the agent appeared in before the current game state in the current game; in other embodiments, the historical game states corresponding to the game state can specifically be the previous n game states that the agent appeared in before the current game state in the current game, where n can be set according to the actual situation, such as 10.

[0285] In this historical state sequence, each game state can be sorted according to the time it appears.

[0286] 204. The server performs state encoding processing on each state in the historical state sequence to obtain the state encoding information of each state in the historical state sequence.

[0287] Specifically, state coding can be divided into two coding methods: fixed coding and learnable coding.

[0288] Fixed coding, also known as hard coding, is a relatively simple and highly interpretable coding method. The coding functions in fixed coding can be directly set by domain experts based on experience. For example, in racing games, to enable an agent to learn obstacle avoidance capabilities, state coding can be based on the agent's corresponding coordinate position. Specifically, besides coordinate coding, fixed coding can also use other discriminative coding methods, such as coding the state based on whether the agent is blocked by an obstacle.

[0289] The learning encoding function does not require prior knowledge from domain experts. Instead, the encoding function is defined as a parameterized encoding function, which can be a neural network or other types. Specifically, the loss function can be defined according to the requirements of the encoding space, and the encoding function can be learned on a sufficient dataset. The advantage of this approach is that it can learn discriminative encoding without prior knowledge.

[0290] Optionally, in this embodiment, the action type of the intelligent agent's game action includes movement action type;

[0291] The step "perform state encoding processing on each state in the historical state sequence to obtain state encoding information for each state in the historical state sequence" may include:

[0292] Obtain the position information of the agent in the target game under each state of the historical state sequence;

[0293] Based on the location information, determine the state encoding information of each state in the historical state sequence.

[0294] Optionally, in this embodiment, the step "performing state encoding processing on each state in the historical state sequence to obtain state encoding information for each state in the historical state sequence" may include:

[0295] The state coding model is used to perform state coding processing on each state in the historical state sequence to obtain the state coding information of each state in the historical state sequence.

[0296] The state coding model can be a neural network model or a tree model; this embodiment does not impose any specific restrictions on it.

[0297] 205. The server classifies the status coding information of each state to determine the proportion of each state under each preset status type.

[0298] In some embodiments, the state coding information of each state is classified and processed. Specifically, based on the state coding information of each state, the preset state type to which each state belongs is determined. Then, the number of states under each preset state type is counted. Finally, based on the total number of states in the historical state sequence and the number of states under each preset state type, the occurrence ratio information of states under each preset state type is determined. The preset state type to which each state belongs can be determined based on the similarity between the state coding information of each state and the preset state coding information corresponding to each preset state type.

[0299] 206. The server calculates the type information entropy of the historical state sequence based on the occurrence ratio information of each preset state type, and obtains the state information entropy of the agent at the corresponding moment of the game state.

[0300] 207. The server determines the exploration reward information corresponding to the agent in the game state based on the state information entropy.

[0301] There are several ways to determine reward information. Specifically, rewards can be divided into hard rewards and soft rewards. Hard rewards are given based on whether the entropy value meets a preset threshold, while soft rewards are given based on the entropy value and a preset increasing function, with the function value of the entropy serving as the reward.

[0302] Optionally, in this embodiment, the step "determining the exploration reward information corresponding to the agent in the game state based on the state information entropy" may include:

[0303] When the state information entropy is greater than a preset threshold, the exploration reward information corresponding to the agent in the game state is determined as the first exploration reward information;

[0304] When the state information entropy is not greater than a preset threshold, the exploration reward information corresponding to the agent in the game state is determined to be the second exploration reward information, and the second exploration reward information is less than the first exploration reward information.

[0305] It is understandable that, in addition to calculation using a single threshold, hard rewards can also be calculated using multiple thresholds, and this embodiment does not impose any restrictions on this.

[0306] Optionally, in this embodiment, the step "determining the exploration reward information corresponding to the agent in the game state based on the state information entropy" may include:

[0307] Get the preset increment function;

[0308] The state information entropy is processed by the preset incrementing function to obtain the exploration reward information corresponding to the game state of the intelligent agent.

[0309] The game strategy model training method of this application can be based on exploration reward information for model training. In this way, during the training process, in order to obtain more rewards, the agent will explore areas with higher entropy values, thereby enhancing the agent's curiosity and improving the agent's obstacle avoidance ability.

[0310] 208. Based on the exploration reward information and the agent's action execution strategy, the server updates the parameters of the game strategy model to obtain a target game strategy model. The action execution strategy includes the game state and game action of the agent each time it performs an action. The target game strategy model is used to control the agent's game actions during the game of the target game.

[0311] The final target game strategy model can improve the obstacle avoidance ability of the intelligent agent, enabling it to have a strong ability to pass levels with different obstacle placements.

[0312] Optionally, in this embodiment, the step "updating the parameters of the game strategy model based on the exploration reward information and the agent's action execution strategy to obtain the target game strategy model" may include:

[0313] Determine the associated game states corresponding to each game state of the agent in this game;

[0314] For each game state, based on the exploration reward information corresponding to the game state and the exploration reward information corresponding to the associated game state, the action reward and penalty information corresponding to the game state is determined.

[0315] Based on the action execution strategy of the agent and the action reward and punishment information corresponding to each game state of the agent in this game, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0316] Specifically, the associated game state of a certain game state can be the state corresponding to the execution of a certain game action based on that game state. The associated game state is the next game state of that game state in the current game.

[0317] Specifically, the reward / penalty information for a particular game state refers to the reward or penalty information for game actions performed based on that game state. This information can be determined by the difference between the exploration reward information of the associated game states and the exploration reward information of the current game state. If the exploration reward information of the next game state is smaller than that of the current game state, it indicates that the game action performed based on that state has reduced the state information entropy, meaning the exploration aspect of that action is lower, and therefore a penalty is imposed on that action.

[0318] Optionally, in this embodiment, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0319] The step "updating the parameters of the game strategy model based on the agent's action execution strategy and the action reward and penalty information corresponding to each game state of the agent in this game, to obtain the target game strategy model" may include:

[0320] The action execution strategy of the agent is parsed to obtain the action data sequence of the agent. The action data sequence includes each strategy data pair arranged by execution time in the current game. The strategy data pair includes the game state and game action at the corresponding execution time.

[0321] For each strategy data pair, determine the target game state in the game strategy model that matches the strategy data pair;

[0322] Based on the action reward and punishment information corresponding to the game state in the strategy data pair, the probability distribution information of the candidate game actions corresponding to the target game state is updated to obtain the target game strategy model.

[0323] This embodiment can encourage the agent to perform game actions with high entropy values ​​through action reward and punishment information, thereby encouraging the agent's curiosity, improving the agent's exploration ability, and thus enhancing the agent's obstacle avoidance ability.

[0324] Optionally, in this embodiment, the step "updating the parameters of the game strategy model based on the exploration reward information and the agent's action execution strategy to obtain the target game strategy model" may include:

[0325] Based on the exploration reward information and the action execution strategy of the agent, the parameters of the game strategy model are updated to obtain the updated game strategy model.

[0326] Based on the state information entropy corresponding to each game state of the agent in the target game, the updated game strategy model is processed to evaluate the agent's obstacle avoidance ability.

[0327] When the updated game strategy model passes the evaluation, it is determined as the target game strategy model.

[0328] In some embodiments, the obstacle avoidance ability of the intelligent agent can be evaluated based on the state information entropy of each game state of the intelligent agent. If the state information entropy of the game state of the intelligent agent is low, it means that the intelligent agent may be stuck in one place and has poor obstacle avoidance ability.

[0329] In other embodiments, the obstacle avoidance ability of the agent can also be evaluated directly by detecting the agent's performance in the target game. For example, if there is an obstacle in the middle of the game map, and the agent moves around the obstacle to either side, its obstacle avoidance ability can be considered relatively strong. If the agent remains stuck at the obstacle, it indicates that its obstacle avoidance ability is relatively weak.

[0330] This application provides a curiosity-guided obstacle avoidance learning method for intelligent agents. Specifically, it proposes training an intelligent agent with level-clearing capabilities to further enhance its obstacle avoidance abilities. During training, historical game state information is encoded and used as features, enabling the agent to perceive state sequence information. Simultaneously, based on the encoded historical game state information, the entropy value of the current game state and historical game state information is calculated. Rewards are given based on the entropy value, encouraging the agent's curiosity and prompting it to move towards states with higher entropy values. This allows the trained agent to navigate obstacles in different levels with varying obstacle placements, based on its historical game state, thus improving its obstacle avoidance capabilities. This method allows for the training of an agent with obstacle avoidance abilities without requiring a large number of levels with different obstacles. This application can be applied to game AI to enhance its obstacle avoidance capabilities.

[0331] As can be seen from the above, this embodiment can obtain the game strategy model of the target game through the server. The game strategy model is used to determine the game actions to be performed by the agent in different game states. Based on the game state of the agent, the game strategy model is used to control the agent's game actions in the target game to complete the target game. The historical state sequence corresponding to the game state is determined. The historical state sequence includes at least one historical game state corresponding to the game state and the game state. The historical game state is the state in which the agent appeared before the current game state. State encoding processing is performed on each state in the historical state sequence to obtain the state encoding information of each state in the historical state sequence. The state encoding information of the states is classified to determine the proportion of states appearing under each preset state type; based on the proportion of states appearing under each preset state type, the type information entropy of the historical state sequence is calculated to obtain the state information entropy of the agent at the corresponding moment of the game state; based on the state information entropy, the exploration reward information corresponding to the agent in the game state is determined; based on the exploration reward information and the agent's action execution strategy, the parameters of the game strategy model are updated to obtain the target game strategy model. The action execution strategy includes the game state and game action when the agent performs an action each time. The target game strategy model is used to control the agent's game actions during the game of the target game.

[0332] This application can combine the state information entropy of the agent's game state with the agent's exploration ability during the training process of the game strategy model, and give exploration rewards based on the agent's state information entropy to encourage the agent's curiosity. This can improve the accuracy and reliability of the game action generation of the trained target game strategy model, more accurately identify obstacles in the game process, and thus better control the agent to avoid obstacles in levels with different obstacle placements, thereby improving the agent's obstacle avoidance ability and enhancing the agent's ability to complete levels.

[0333] To better implement the above methods, embodiments of this application also provide a game strategy model training device, such as... Figure 3 As shown, the game strategy model training device may include a control unit 301, a calculation unit 302, a reward determination unit 303, and an update unit 304, as follows:

[0334] (1) Control unit 301;

[0335] The control unit is used to control the agent's game actions in the target game based on the agent's game state using a game strategy model of the target game. The game strategy model is used to determine the game actions that the agent should perform in different game states.

[0336] Optionally, in some embodiments of this application, the game strategy model training device further includes an acquisition unit, which is used to acquire the game strategy model of the target game; specifically, the acquisition unit may include a model acquisition subunit, an action control subunit, and an initial model update subunit, as follows:

[0337] The model acquisition subunit is used to acquire the initial game strategy model of the target game. The initial game strategy model is used to determine the game actions that the agent should perform in different game states.

[0338] An action control subunit is used to control the game actions of the agent in the target game based on the game state of the agent in multiple games of the target game, using the initial game strategy model.

[0339] The initial model update subunit is used to update the parameters of the initial game strategy model based on the game state and game actions of the agent each time it performs an action that meets the preset level-clearing conditions, so as to obtain the game strategy model. The agent under the control of the game strategy model has the ability to clear the level.

[0340] Optionally, in some embodiments of this application, the initial model update subunit may be specifically used to calculate the completion rate of the agent in the target game based on the game result data of the agent in each game; when the completion rate does not meet the preset conditions, the parameters of the initial game strategy model are updated based on the game state and game actions of the agent that meets the preset completion conditions each time it performs an action, to obtain an updated initial game strategy model; and return to execute the step of controlling the agent's game actions in the target game based on the agent's game state in multiple games of the target game through the initial game strategy model, until the completion rate of the agent in the target game meets the preset conditions, to obtain the game strategy model.

[0341] Optionally, in some embodiments of this application, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0342] The control unit may include a first determining subunit, a second determining subunit, and a control subunit, as follows:

[0343] The first determining subunit is used to input the game state of the agent into the game strategy model of the target game to determine at least one candidate game action of the agent;

[0344] The second determining subunit is used to determine the target game action to be performed by the agent from the candidate game actions by using the probability distribution information of the game strategy model;

[0345] A control subunit is used to control the agent to perform target game actions in the target game.

[0346] (2) Calculation unit 302;

[0347] The computing unit is used to calculate the state information entropy of the agent at the time corresponding to the game state based on the game state and the historical game state, wherein the historical game state is the state in which the agent appeared before the game state in the current game.

[0348] Optionally, in some embodiments of this application, the computing unit includes a sequence determination subunit, an encoding subunit, a classification subunit, and an entropy calculation subunit, as follows:

[0349] The sequence determination subunit is used to determine the historical state sequence corresponding to the game state, and the historical state sequence includes at least one historical game state corresponding to the game state and the game state.

[0350] The encoding subunit is used to perform state encoding processing on each state in the historical state sequence to obtain the state encoding information of each state in the historical state sequence;

[0351] The classification subunit is used to classify the state coding information of each state in order to determine the proportion of occurrence of each state under each preset state type.

[0352] The entropy calculation subunit is used to calculate the type information entropy of the historical state sequence based on the occurrence ratio information of each preset state type, so as to obtain the state information entropy of the agent at the corresponding moment of the game state.

[0353] Optionally, in some embodiments of this application, the action type of the intelligent agent's game actions includes movement action types;

[0354] Specifically, the encoding subunit can be used to obtain the position information of the agent in the target game under each state of the historical state sequence; and determine the state encoding information of each state in the historical state sequence based on the position information.

[0355] Optionally, in some embodiments of this application, the encoding subunit may be specifically used to perform state encoding processing on each state in the historical state sequence through a state encoding model to obtain state encoding information of each state in the historical state sequence.

[0356] The game strategy model training device may further include a state coding model training unit, as follows:

[0357] The state encoding model training unit is used to acquire training data, which includes at least one sample game state of the agent in the target game and the expected state encoding information corresponding to the sample game state; the state encoding model is used to perform state encoding processing on the sample game state to obtain the sample state encoding information of the sample game state; the parameters of the state encoding model are adjusted according to the sample state encoding information and the expected state encoding information to obtain the trained state encoding model.

[0358] Optionally, in some embodiments of this application, the game strategy model training device may further include an expected encoding determination unit, as follows:

[0359] The expected encoding determination unit is used to divide the preset encoding space of the game state of the intelligent agent according to the exploration ability requirement information of the intelligent agent, to obtain multiple state encoding spaces, each state encoding space corresponding to a state encoding information; and to determine the expected state encoding information corresponding to the sample game state according to the state encoding space in which the sample game state is located.

[0360] (3) Reward determination unit 303;

[0361] The reward determination unit is used to determine the exploration reward information corresponding to the agent in the game state based on the state information entropy.

[0362] Optionally, in some embodiments of this application, the reward determination unit may include a third determination subunit and a fourth determination subunit, as follows:

[0363] The third determining subunit is used to determine the exploration reward information corresponding to the agent in the game state as the first exploration reward information when the state information entropy is greater than a preset threshold.

[0364] The fourth determining subunit is used to determine the exploration reward information corresponding to the agent in the game state as the second exploration reward information when the state information entropy is not greater than a preset threshold, wherein the second exploration reward information is less than the first exploration reward information.

[0365] Optionally, in some embodiments of this application, the reward determination unit may include a function acquisition subunit and a computation subunit, as follows:

[0366] The function acquisition subunit is used to acquire a preset incrementing function;

[0367] The computational subunit is used to process the state information entropy through the preset incrementing function to obtain the exploration reward information corresponding to the agent in the game state.

[0368] (4) Update unit 304;

[0369] The update unit is used to update the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, so as to obtain the target game strategy model.

[0370] Optionally, in some embodiments of this application, the updating unit may include an association state determination subunit, an action reward / penalty determination subunit, and an updating subunit, as follows:

[0371] The associated state determination subunit is used to determine the associated game states corresponding to each game state of the agent in the current game.

[0372] The action reward and punishment determination subunit is used to determine the action reward and punishment information corresponding to each game state based on the exploration reward information corresponding to the game state and the exploration reward information corresponding to the associated game state.

[0373] The update subunit is used to update the parameters of the game strategy model based on the game state and game action of the agent each time it performs an action, as well as the action reward and punishment information corresponding to each game state of the agent in the current game, to obtain the target game strategy model.

[0374] Optionally, in some embodiments of this application, the game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states;

[0375] The update subunit can be specifically used to construct the action data sequence of the agent. The action data sequence includes various strategy data pairs arranged by execution time in the current game. The strategy data pair includes the game state and game action at the corresponding execution time. For each strategy data pair, a target game state that matches the strategy data pair in the game strategy model is determined. Based on the action reward and penalty information corresponding to the game state in the strategy data pair, the probability distribution information of the candidate game actions corresponding to the target game state is updated to obtain the target game strategy model.

[0376] Optionally, in some embodiments of this application, the updating unit may include a reward / penalty determination subunit, a parameter update subunit, a fifth determination subunit, and a return execution subunit, as follows:

[0377] The reward and punishment determination subunit is used to determine the action reward and punishment information corresponding to the current game state based on the exploration reward information corresponding to the current game state of the agent and the exploration reward information of the game state corresponding to the current game state after the agent performs the current game action;

[0378] The parameter update subunit is used to update the parameters of the game strategy model based on the action reward and punishment information of the current game state, the current game state, and the game actions performed by the agent in the current game state.

[0379] The fifth determining subunit is used to take the game state after the agent performs the game action as the new current game state;

[0380] The execution subunit is returned to control the agent's game actions in the target game based on the updated game strategy model and the agent's new current game state. The execution subunit then returns to the step of calculating the state information entropy of the agent at the corresponding moment in the game state based on the game state and historical game states, so as to obtain the target game strategy model.

[0381] Optionally, in some embodiments of this application, the updating unit may include a model updating subunit, an evaluation subunit, and a sixth determining subunit, as follows:

[0382] The model update subunit is used to update the parameters of the game strategy model based on the exploration reward information and the game state and game action of the agent each time it performs an action, so as to obtain the updated game strategy model.

[0383] The evaluation subunit is used to evaluate the obstacle avoidance ability of the agent based on the state information entropy corresponding to each game state in the target game.

[0384] The sixth determining subunit is used to determine the updated game strategy model as the target game strategy model when the updated game strategy model passes the evaluation.

[0385] Optionally, in some embodiments of this application, the game strategy model training device may further include a model application unit, as follows:

[0386] The model application unit is used to acquire the current game state data of the agent in the target game; input the current game state data of the agent into the target game strategy model, so as to control the agent's game actions in the target game based on the current game state data through the target game strategy model; and issue game completion rewards to the agent when the agent meets the preset completion conditions of the target game.

[0387] As can be seen from the above, in this embodiment, the control unit 301 controls the game actions of the agent in the target game based on the game state of the agent through the game strategy model of the target game. The game strategy model is used to determine the game actions to be performed by the agent in different game states. The calculation unit 302 calculates the state information entropy of the agent at the corresponding moment of the game state based on the game state and the historical game state. The historical game state is the state of the agent before the current game state. The reward determination unit 303 determines the exploration reward information corresponding to the agent in the game state based on the state information entropy. The update unit 304 updates the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, thereby obtaining the target game strategy model.

[0388] This application can combine the state information entropy of the agent's game state with the agent's exploration ability during the training process of the game strategy model, and give exploration rewards based on the agent's state information entropy to encourage the agent's curiosity. This can improve the accuracy and reliability of the game action generation of the trained target game strategy model, more accurately identify obstacles in the game process, and thus better control the agent to avoid obstacles in levels with different obstacle placements, thereby improving the agent's obstacle avoidance ability and enhancing the agent's ability to complete levels.

[0389] This application also provides an electronic device, such as... Figure 4 The diagram shows a structural schematic of an electronic device involved in an embodiment of this application. This electronic device can be a terminal or a server, specifically:

[0390] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0391] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0392] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0393] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0394] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0395] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0396] Using a game strategy model of the target game, and based on the agent's game state, the game actions of the agent in the target game are controlled. The game strategy model is used to determine the game actions to be performed by the agent in different game states. Based on the game state and historical game states, the state information entropy of the agent at the corresponding moment of the game state is calculated. The historical game states are the states that the agent appeared in before the current game state in this game. According to the state information entropy, the exploration reward information corresponding to the agent in the game state is determined. Based on the exploration reward information and the game state and game actions of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0397] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0398] As can be seen from the above, this embodiment can control the game actions of the agent in the target game based on the game state of the agent through the game strategy model of the target game. The game strategy model is used to determine the game actions to be performed by the agent in different game states. Based on the game state and the historical game state, the state information entropy of the agent at the corresponding moment of the game state is calculated. The historical game state is the state of the agent before the current game state. According to the state information entropy, the exploration reward information corresponding to the agent in the game state is determined. Based on the exploration reward information and the game state and game action of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0399] This application can combine the state information entropy of the agent's game state with the agent's exploration ability during the training process of the game strategy model, and give exploration rewards based on the agent's state information entropy to encourage the agent's curiosity. This can improve the accuracy and reliability of the game action generation of the trained target game strategy model, more accurately identify obstacles in the game process, and thus better control the agent to avoid obstacles in levels with different obstacle placements, thereby improving the agent's obstacle avoidance ability and enhancing the agent's ability to complete levels.

[0400] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0401] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the game strategy model training methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0402] Using a game strategy model of the target game, and based on the agent's game state, the game actions of the agent in the target game are controlled. The game strategy model is used to determine the game actions to be performed by the agent in different game states. Based on the game state and historical game states, the state information entropy of the agent at the corresponding moment of the game state is calculated. The historical game states are the states that the agent appeared in before the current game state in this game. According to the state information entropy, the exploration reward information corresponding to the agent in the game state is determined. Based on the exploration reward information and the game state and game actions of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the target game strategy model.

[0403] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0404] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0405] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the game strategy model training methods provided in the embodiments of this application, the beneficial effects that any of the game strategy model training methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0406] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the game strategy model training described above.

[0407] The above provides a detailed description of a game strategy model training method and related equipment provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a game strategy model, characterized in that, include: Based on the game state of the agent, the game strategy model of the target game is used to control the game actions of the agent in the target game. The game strategy model is used to determine the game actions that the agent should perform in different game states. Based on the game state and the historical game state, calculate the state information entropy of the agent at the time corresponding to the game state. The historical game state is the state that the agent appeared in before the game state in the current game. Based on the state information entropy, the exploration reward information corresponding to the agent in the game state is determined; Based on the exploration reward information and the game state and actions of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the target game strategy model.

2. The method according to claim 1, characterized in that, The game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states; The method of controlling the agent's game actions in the target game based on the agent's game state using the game strategy model of the target game includes: The game state of the agent is input into the game strategy model of the target game to determine at least one candidate game action of the agent; The target game action to be performed by the agent is determined from the candidate game actions by using the probability distribution information of the game strategy model. Control the intelligent agent to perform target game actions in the target game.

3. The method according to claim 1, characterized in that, The step of calculating the state information entropy of the agent at the corresponding moment in the game state based on the game state and historical game states includes: Determine the historical state sequence corresponding to the game state, wherein the historical state sequence includes at least one historical game state corresponding to the game state and the game state; Each state in the historical state sequence is processed by state encoding to obtain the state encoding information of each state in the historical state sequence; The state coding information of each state is classified and processed to determine the proportion of occurrence of each state under each preset state type; Based on the occurrence percentage information of each preset state type, the type information entropy of the historical state sequence is calculated to obtain the state information entropy of the agent at the corresponding moment of the game state.

4. The method according to claim 3, characterized in that, The game actions of the intelligent agent include movement actions; The step of performing state encoding processing on each state in the historical state sequence to obtain state encoding information for each state in the historical state sequence includes: Obtain the position information of the agent in the target game under each state of the historical state sequence; Based on the location information, determine the state encoding information of each state in the historical state sequence.

5. The method according to claim 3, characterized in that, The step of performing state encoding processing on each state in the historical state sequence to obtain state encoding information for each state in the historical state sequence includes: The state coding model is used to perform state coding processing on each state in the historical state sequence to obtain the state coding information of each state in the historical state sequence. Before performing state encoding processing on each state in the historical state sequence using a state encoding model to obtain the state encoding information of each state in the historical state sequence, the method further includes: Acquire training data, which includes at least one sample game state of the agent in the target game and the expected state encoding information corresponding to the sample game state; The sample game state is processed by a state coding model to obtain the sample state coding information of the sample game state. Based on the sample state encoding information and the desired state encoding information, the parameters of the state encoding model are adjusted to obtain the trained state encoding model.

6. The method according to claim 5, characterized in that, The method further includes: Based on the information regarding the exploration capabilities required by the intelligent agent, the preset encoding space of the game state of the intelligent agent is divided to obtain multiple state encoding spaces, each of which corresponds to a state encoding information. Based on the state encoding space in which the sample game state is located, determine the expected state encoding information corresponding to the sample game state.

7. The method according to claim 1, characterized in that, The step of determining the exploration reward information corresponding to the agent in the game state based on the state information entropy includes: When the state information entropy is greater than a preset threshold, the exploration reward information corresponding to the agent in the game state is determined as the first exploration reward information; When the state information entropy is not greater than a preset threshold, the exploration reward information corresponding to the agent in the game state is determined to be the second exploration reward information, and the second exploration reward information is less than the first exploration reward information.

8. The method according to claim 1, characterized in that, The step of determining the exploration reward information corresponding to the agent in the game state based on the state information entropy includes: Get the preset increment function; The state information entropy is processed by the preset incrementing function to obtain the exploration reward information corresponding to the game state of the intelligent agent.

9. The method according to claim 1, characterized in that, The process of updating the parameters of the game strategy model based on the exploration reward information and the game state and actions of the agent at each execution of an action to obtain the target game strategy model includes: Determine the associated game states corresponding to each game state of the agent in this game; For each game state, based on the exploration reward information corresponding to the game state and the exploration reward information corresponding to the associated game state, the action reward and penalty information corresponding to the game state is determined. Based on the game state and game action of the agent each time it performs an action, as well as the reward and penalty information of the action corresponding to each game state of the agent in this game, the parameters of the game strategy model are updated to obtain the target game strategy model.

10. The method according to claim 9, characterized in that, The game strategy model includes probability distribution information of candidate game actions corresponding to the agent in different game states; The step of updating the parameters of the game strategy model based on the game state and game action of the agent each time it performs an action, and the action reward and penalty information corresponding to each game state of the agent in the current game, to obtain the target game strategy model, includes: Construct the action data sequence of the agent, the action data sequence including each strategy data pair arranged by execution time in the current game, the strategy data pair including the game state and game action at the corresponding execution time; For each strategy data pair, determine the target game state in the game strategy model that matches the strategy data pair; Based on the action reward and punishment information corresponding to the game state in the strategy data pair, the probability distribution information of the candidate game actions corresponding to the target game state is updated to obtain the target game strategy model.

11. The method according to claim 1, characterized in that, The process of updating the parameters of the game strategy model based on the exploration reward information and the game state and actions of the agent at each execution of an action to obtain the target game strategy model includes: Based on the exploration reward information corresponding to the current game state of the agent, and the exploration reward information of the game state corresponding to the current game state after the agent performs the current game action, the action reward and punishment information corresponding to the current game state is determined; The game strategy model is updated with parameters based on the action reward and punishment information of the current game state, the current game state, and the game actions performed by the agent in the current game state. The game state after the agent performs the game action is taken as the new current game state; Using the updated game strategy model, based on the agent's new current game state, the game actions of the agent in the target game are controlled, and the step of calculating the state information entropy of the agent at the corresponding moment of the game state based on the game state and historical game states is returned to execute, so as to obtain the target game strategy model.

12. The method according to claim 1, characterized in that, The process of updating the parameters of the game strategy model based on the exploration reward information and the game state and actions of the agent at each execution of an action to obtain the target game strategy model includes: Based on the exploration reward information and the game state and game actions of the agent each time it performs an action, the parameters of the game strategy model are updated to obtain the updated game strategy model. Based on the state information entropy corresponding to each game state of the agent in the target game, the updated game strategy model is processed to evaluate the agent's obstacle avoidance ability. When the updated game strategy model passes the evaluation, it is determined as the target game strategy model.

13. The method according to claim 1, characterized in that, Before controlling the agent's game actions in the target game based on the agent's game state using the game strategy model of the target game, the method further includes: Obtain the initial game strategy model of the target game, which is used to determine the game actions to be performed by the agent in different game states; Using the initial game strategy model, the agent's game actions in the target game are controlled based on the agent's game state in multiple games of the target game. Based on the game state and actions of the agent each time it performs an action that meets the preset clearance conditions, the parameters of the initial game strategy model are updated to obtain a game strategy model. The agent controlled by the game strategy model has the ability to clear the level.

14. The method according to claim 13, characterized in that, The game strategy model is updated by updating the parameters of the initial game strategy model based on the game state and game actions of the agent each time it performs an action that meets the preset level-clearing conditions, resulting in a game strategy model including: Based on the game result data of the agents in each game, the success rate of the agents in the target game is statistically analyzed. When the pass rate does not meet the preset conditions, the parameters of the initial game strategy model are updated based on the game state and game actions of the agent that meets the preset pass conditions each time it performs an action, so as to obtain the updated initial game strategy model. Return to the step of controlling the agent's game actions in the target game based on the agent's game state in multiple games of the target game using the initial game strategy model, until the agent's completion rate in the target game meets the preset conditions, thus obtaining the game strategy model.

15. The method according to claim 1, characterized in that, After updating the parameters of the game strategy model based on the exploration reward information and the game state and actions of the agent at each execution of an action to obtain the target game strategy model, the method further includes: Obtain the current game state data of the agent in the target game; The current game state data of the agent is input into the target game strategy model so that the target game strategy model can control the agent's game actions in the target game based on the current game state data. When the agent meets the preset completion conditions of the target game, a game completion reward is issued to the agent.

16. A game strategy model training device, characterized in that, include: The control unit is used to control the game actions of the agent in the target game based on the game state of the agent, using the game strategy model of the target game. The game strategy model is used to determine the game actions to be performed by the agent in different game states. The computing unit is used to calculate the state information entropy of the agent at the time corresponding to the game state based on the game state and the historical game state, wherein the historical game state is the state of the agent in the current game before the game state. The reward determination unit is used to determine the exploration reward information corresponding to the agent in the game state based on the state information entropy. The update unit is used to update the parameters of the game strategy model based on the exploration reward information and the game state and game actions of the agent each time it performs an action, so as to obtain the target game strategy model.

17. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program within the memory to perform the operations in the game strategy model training method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps in the game strategy model training method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps in the game strategy model training method according to any one of claims 1 to 15.

Images