Model training method and apparatus, and computer device, computer-readable storage medium and computer program product

Abstract

Provided in the present application are a model training method and apparatus, and a computer device, a computer-readable storage medium and a computer program product. The embodiments of the present application can be applied to various scenes such as games, artificial intelligence, intelligent transportation and assisted driving. The method comprises: acquiring state data of a first agent in a virtual scene, and a first prediction model to be trained that corresponds to the first agent; using said first prediction model to predict state data, so as to obtain an action instruction set including a path-finding mode; on the basis of the action instruction set, controlling the first agent to execute corresponding actions, so as to obtain execution results; and on the basis of reward information of the plurality of execution results, adjusting parameters of said first prediction model, so as to obtain a trained first prediction model, wherein the reward information is determined on the basis of the plurality of execution results, a main reward coefficient and an assistance reward coefficient.

Description

Model training method and device, computer device, computer readable storage medium, and computer program product

[0001] Cross-reference to Related Applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202410545713.X, filed on April 30, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, and in particular to a model training method and device, a computer device, a computer readable storage medium, and a computer program product. BACKGROUND

[0004] With the development of artificial intelligence technology, the application prospect of game agents in the game field is also becoming more and more broad. Game agents are one of the important elements in games, which can add depth, interest and interactivity to games, and improve the game experience and participation of players.

[0005] In the related art, the first way is to control the pathfinding mode of the game agent based on a navigation mesh and a scene query system, and the second way is to control the pathfinding mode of the game agent based on atomic actions. The above related technologies all use a single and fixed pathfinding mode, which has the problem of poor training effect, and finally leads to the failure of model online deployment, poor strategy effect learned by the model, or even the failure of convergence. SUMMARY

[0006] The embodiments of the present application provide a model training method, device, computer device, computer readable storage medium, and computer program product, which can improve the efficiency and accuracy of model training.

[0007] The technical solutions of the embodiments of the present application are implemented as follows:

[0008] The embodiments of the present application provide a model training method applied to a computer device, which comprises the following steps:

[0009] Obtaining state data of a first agent in a virtual scene and a first prediction model to be trained corresponding to the first agent; the first prediction model to be trained is used to control the first agent to move based on the state data of the first agent;

[0010] Using the first prediction model to be trained to predict the state data to obtain a set of action instructions, the set of action instructions comprising a pathfinding mode and a plurality of action information;

[0011] The first agent is controlled to move based on the pathfinding mode, and corresponding actions are performed based on each of the action information, to obtain execution results.

[0012] The parameters of the first trained prediction model are adjusted based on reward information of a plurality of execution results, to obtain a trained first prediction model, wherein the reward information is determined based on the plurality of execution results, a main reward coefficient, and an auxiliary reward coefficient.

[0013] An embodiment of the present application provides a model training device, and the device comprises:

[0014] A first acquisition module is configured to acquire state data of a first agent in a virtual scene and a first trained prediction model corresponding to the first agent; the first trained prediction model is used to control the first agent to move based on the state data of the first agent.

[0015] An instruction prediction module is configured to predict the state data by using the first trained prediction model, to obtain a set of action instructions, wherein the set of action instructions comprises a pathfinding mode and a plurality of action information.

[0016] An execution module is configured to control the first agent to move based on the pathfinding mode, and perform corresponding actions based on each of the action information, to obtain execution results.

[0017] An adjustment module is configured to adjust parameters of the first trained prediction model based on reward information of a plurality of execution results, to obtain a trained first prediction model, wherein the reward information is determined based on the plurality of execution results, a main reward coefficient, and an auxiliary reward coefficient.

[0018] An embodiment of the present application provides a computer device, and the computer device comprises:

[0019] A memory is configured to store computer executable instructions.

[0020] A processor is configured to execute the computer executable instructions stored in the memory, to implement a model training method provided by an embodiment of the present application.

[0021] An embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and is used to cause a processor to execute the model training method provided by an embodiment of the present application.

[0022] An embodiment of the present application provides a computer program product, which comprises a computer program or computer executable instructions, and the computer program or computer executable instructions are executed by a processor to implement the model training method provided by an embodiment of the present application.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] In the embodiments of the present application, the game session includes a first agent and a second agent that are in mutual confrontation, and the first agent corresponds to the first prediction model to be trained. Then, the state data of the first agent can be predicted by using the first prediction model to be trained to obtain a set of action instructions containing a pathfinding mode. Since the first prediction model to be trained can control the first agent to move according to different pathfinding modes based on different scenes in which the first agent is located, when the first agent is in different virtual scenes, a set of action instructions containing different pathfinding modes can be predicted, and the pathfinding mode in the set of action instructions matches the scene in which the first agent is located, thereby realizing the diversification of the pathfinding mode. Then, the first agent is controlled to perform corresponding actions based on the set of action instructions to obtain an execution result. Finally, the rewards of the multiple execution results are determined based on the multiple execution results, the main reward coefficient and the auxiliary reward coefficient, and the parameters of the first prediction model to be trained are adjusted based on the rewards of the multiple execution results to obtain the trained first prediction model. In this way, the mixed pathfinding and movement of the first agent can be realized, the pathfinding mode and the movement mode are diversified, the training effect of the first prediction model to be trained is improved through the diversified pathfinding mode, and the training accuracy of the first prediction model to be trained is also improved since the pathfinding mode of the first agent matches the scene in which the first agent is located. In addition, since the main reward coefficient and the auxiliary reward coefficient are used simultaneously when determining the rewards, the learning efficiency and performance of the first agent in the strategy learning task can be improved, that is, the prediction performance of the trained first prediction model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 is a schematic diagram of a network architecture of a model training system 100 provided by an embodiment of the present application;

[0026] FIG. 2 is a schematic diagram of the structure of a server 400 provided by an embodiment of the present application;

[0027] FIG. 3 is a first flowchart of a model training method provided by an embodiment of the present application;

[0028] FIG. 4 is a first flowchart of a prediction method provided by an embodiment of the present application;

[0029] FIG. 5 is a flowchart of a feature extraction method provided by an embodiment of the present application;

[0030] FIG. 6 is a second flowchart of a prediction method provided by an embodiment of the present application;

[0031] FIG. 7 is a third flowchart of a prediction method provided by an embodiment of the present application;

[0032] FIG. 8 is a fourth flowchart of a prediction method according to an embodiment of the present application;

[0033] FIG. 9 is a flowchart of constructing a set of action instructions according to an embodiment of the present application;

[0034] FIG. 10 is a flowchart of performing an action according to an embodiment of the present application;

[0035] FIG. 11 is a flowchart of adjusting parameters according to an embodiment of the present application;

[0036] FIG. 12 is a second flowchart of a model training method according to an embodiment of the present application;

[0037] FIG. 13 is an interface display diagram of a farm map according to an embodiment of the present application;

[0038] FIG. 14A is a flowchart of determining a battle configuration logic according to an embodiment of the present application;

[0039] FIG. 14B is an interface display diagram of presenting birth coordinates according to an embodiment of the present application.

[0040] FIG. 15 is a general structure diagram of a model training stage according to an embodiment of the present application;

[0041] FIG. 16 is a structure diagram of a synchronous inference mode according to an embodiment of the present application;

[0042] FIG. 17 is a structure flowchart of an asynchronous inference mode according to an embodiment of the present application;

[0043] FIG. 18A is a structure diagram of a network structure of a hybrid pathfinding action head according to an embodiment of the present application;

[0044] FIG. 18B is a structure diagram of a time slice according to an embodiment of the present application;

[0045] FIG. 19 is a flowchart of a hybrid pathfinding according to an embodiment of the present application;

[0046] FIG. 20 is a framework structure diagram of a model structure according to an embodiment of the present application;

[0047] FIG. 21A is a distribution diagram of an active area of an AI trained based on a hybrid mobile pathfinding scheme in a map according to an embodiment of the present application;

[0048] FIG. 21B is a distribution diagram of an active area of an AI trained based on an atomic mobile pathfinding scheme in a map according to an embodiment of the present application;

[0049] FIG. 22 is an AI battle behavior tree win rate result display diagram according to an embodiment of the present application;

[0050] FIG. 23 is a schematic diagram of an AI battle behavior tree tie rate result display according to an embodiment of the present application.

[0051] It should be noted that the above-mentioned "first", "second" are only used to distinguish different schemes, and do not represent the advantages or disadvantages of the schemes or the priority in the implementation process. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those skilled in the art without making creative labor are within the scope of protection of the present application.

[0053] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0054] In the following description, the terms "first / second / third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first / second / third" can interchange the specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuitry or memory) or a combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0056] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0057] In the embodiments of the present application, the relevant data collection process should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0058] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0059] 1) Intelligent agents: In the gaming field, intelligent agents typically refer to virtual characters or agent characters within a game. They are controlled by computer programs and exhibit behavioral characteristics that simulate human intelligence. These intelligent agents can be enemies, allies, non-player characters (NPCs), or other roles in the game. The design and implementation of intelligent agents can enhance the playability, challenge, and realism of a game.

[0060] 2) Game agents: In a game scenario, the machine that performs reinforcement learning is called an agent. It is a game artificial intelligence (AI) driven by neural networks. The environment that interacts with the agent is called the environment. The game agent makes reasonable decisions by analyzing the game environment. It can cooperate and compete with other agents or players, making it behave like a human.

[0061] 3) Reinforcement Learning (RL) is a machine learning method that aims to enable an agent to learn how to make decisions through interaction with its environment in order to maximize cumulative rewards. In reinforcement learning, the agent explores the environment by trying different actions and adjusts its behavior based on reward signals from the environment to maximize long-term cumulative rewards. This learning method is similar to trial-and-error learning, where the agent optimizes its behavioral strategies through continuous experimentation and feedback. Reinforcement learning is often used to solve problems requiring dynamic decision-making, such as training game agents, robot control, and autonomous driving.

[0062] 4) Supervised learning (SL) is one of the paradigms of machine learning. It is a data analysis method that iteratively trains an algorithm from a labeled dataset to enable the model to accurately classify data or predict results.

[0063] 5) Navigation Mesh (NavMesh) is the walking surface for navigation and pathfinding in complex spaces. It is a two-dimensional polygonal mesh data structure that defines which areas of the environment the game agent can traverse. Within these areas, the game agent can move freely without being hindered by environmental obstacles. When the game agent receives a command, the navigation mesh will automatically calculate the shortest reachable path to the target point, helping the game agent to find its way in complex airspace and easily achieving the automatic pathfinding effect of AI.

[0064] 6) Environment Query System (EQS), a feature of UE4 AI system, is used to collect data from the environment and query for information that meets certain conditions in the current scene, then divide the weights and return the results according to the weights, so that the AI can respond differently based on the collected results.

[0065] 7) Depth Map (DM), a two-dimensional data structure that takes the distance (depth) from the image collector to each point in the scene as the pixel value, can accurately depict the pure depth information in front of the game agent. The value closer to 1 indicates that the front is brighter and the field of view is more transparent.

[0066] Depth map refers to an image where each pixel's value represents the distance from the observer (usually a camera) to the corresponding point in the scene. It is usually presented in the form of a grayscale image, where darker pixels represent farther distances, and brighter pixels represent closer distances. In game development, depth maps are often used to achieve depth of field effects, simulating the change in depth of field caused by different camera focal lengths. By using depth maps, game engines can determine which objects are within the focus range and which objects are within the blur range, resulting in a more realistic visual effect.

[0067] 8) Ring Ray, a way of environmental perception, is a type of ray that emanates from a point and extends outward in all directions around it, forming a circular or ring-shaped array of rays. It is used to describe the terrain information around the obstacle.

[0068] Ring Ray is a technique commonly used in computer graphics and game development to detect collisions between objects or determine the distance between objects. Unlike regular rays, which extend in a single direction from a point, ring rays are emitted in all directions from a point, forming a circular or ring-shaped array of rays.

[0069] 9) Original Action (OA), refers to the most basic and fine-grained primitive action that a game character operates in a game. This type of action cannot be further decomposed and refined. In concurrent programming, original actions are often used to ensure synchronization and consistency between multiple threads or processes.

[0070] 10) Behavior Tree, a tree-like structure used to control NPC decision-making in games. It is a graphical tool and method used to describe and control the behavior of game characters or agents. It is a tree structure composed of a series of nodes, each representing a behavior, condition or control logic. Behavior trees are widely used in game development, artificial intelligence and robotics.

[0071] 11) Proximal Policy Optimization (PPO), is an online deep reinforcement learning algorithm based on policy gradient optimization for continuous or discrete action space. The core idea of PPO is to control the magnitude of policy change at each update to keep the update within a controllable range, thereby improving the stability of the algorithm.

[0072] 12) Auxiliary rewards, in reinforcement learning algorithms, are designed to help agents achieve different types of differentiated performance and goals. Auxiliary rewards are usually additional reward signals that help the agent learn specific aspects of the task or achieve specific goals during training. Auxiliary rewards are often used in conjunction with the main task rewards to help speed up the learning process, improve performance, and in some cases, help solve difficult problems in the task.

[0073] Common auxiliary rewards include exploration rewards, time rewards, distance rewards, cover-seeking rewards, action diversity rewards, etc., which can be used in different learning frameworks such as reinforcement learning and supervised learning. By designing and using auxiliary rewards reasonably, the learning efficiency and performance of the agent in the task can be improved.

[0074] 13) Player versus Player (PVP), in this mode, players can directly compete, compete or cooperate with other players, rather than interacting with the game's artificial intelligence (AI) or computer-controlled characters. PVP mode is very common in many games, including e-sports games, Massive Multiplayer Online (MMO), shooting games, fighting games, and various types of games. PVP mode usually increases the competitiveness and challenge of the game, and is one of the favorite game content for many players.

[0075] 14) Player versus Environment (PVE), in this mode, players interact and compete with the environment, tasks, monsters or other non-player characters (NPCs) in the game, rather than directly competing with other players. PVE mode usually includes activities such as players completing tasks, exploring maps, defeating enemies, exploring instances or dungeons, etc. In this mode, players usually need to overcome various challenges in the game to improve their skills and equipment to achieve the game's set goals or obtain rewards. PVE mode is very common in many single-player games and multiplayer online games.

[0076] 15) Embedding, in essence, is "compressing data" to describe higher-dimensional n-dimensional features with lower k-dimensional features. The information compression process usually loses information, including the loss of redundant information and the loss of some useful information. Embedding can be seen as the parameter matrix of the one-hot layer in the neural network.

[0077] In the process of training an agent, "embedding" usually refers to the process of mapping high-dimensional discrete data (such as categorical features, symbolic data) to a low-dimensional continuous vector space. This mapping can be achieved through the embedding layer in the neural network.

[0078] In the process of training an agent, embedding is commonly used to process various types of input data, including word embedding, categorical embedding, action embedding, and state embedding.

[0079] By using embedding technology, the original discrete data can be converted into continuous vector representation, improving the data representation ability and the generalization ability of the model, thereby improving the effect and performance of training the agent.

[0080] 16) Action Mask (AM), in deep reinforcement learning training, when the network infers the action distribution, "unreasonable or unexecutable action dimensions" are temporarily shielded, and then the action is sampled from "reasonable or executable actions" according to the original probability. This ensures that the inferred action can obtain valid data samples in the game environment, avoiding wasting computing resources due to returning invalid actions.

[0081] Action mask is used to limit the set of available actions when an agent selects an action. Action mask can ensure that the agent only selects from allowed actions and ignores those that are not available, making the decision-making process of the agent more efficient and accurate. Action mask can be represented by a binary vector (or mask), where each element corresponds to a possible action. If the action is allowed, the mask value of the corresponding bit is 1; if the action is not allowed, the mask value of the corresponding action is 0. For example, assuming there are 5 possible actions, the action mask can be represented as: [1, 0, 1, 0, 0]. This means that the agent can execute the first and third actions, while other actions are not executable.

[0082] 17) Zero-sum game, also known as zero-sum game, is a concept of game theory, which means that the sum of the interests of all parties to the game is zero, or a constant, that is, one party gains, and the other party loses. In a zero-sum game, all parties are uncooperative, which is a non-cooperative game. If all parties to the game make different strategy combinations so that all participants benefit or suffer, it is a non-zero-sum game, and there is a possibility of cooperation among all parties in a non-zero-sum game.

[0083] 18) Best response (BR), is the strategy that a player takes when other players choose a fixed strategy, which produces the most favorable result for himself.

[0084] 19) Self-play (SP), is an iterative process to solve the best response (BR), the agent uses its own (i.e. strategy) previous copy as an opponent to train and update the current strategy, and continues to iterate, by facing an increasingly enhanced opponent to train itself, it can prevent a huge capability gap from making the strategy learn nothing, thereby effectively increasing training efficiency and continuously improving model capability. This training method is very effective in zero-sum games such as chess and Go (AlphaGo), but it needs to meet two conditions: the model capability is improved in each round, and the game environment is a transitive game environment.

[0085] 20) Nash equilibrium (NE), also known as non-cooperative game equilibrium, is an important concept of game theory. In a game process, for each player, when other players do not change their strategies, the strategies of all players are the best response. At this time, no player can obtain greater benefits by changing their own strategy while keeping the strategies of the remaining players unchanged. Then it is called that the current joint strategy has reached Nash equilibrium. Nash equilibrium is not necessarily the optimal solution of all parties, it expresses that the current game environment reaches a balanced and stable state.

[0086] In order to better understand the model training method provided by the embodiments of the present application, first, the model training method in the related art and the existing shortcomings are described.

[0087] There are mainly two kinds of model training methods in the related art:

[0088] Related technology 1, model training method based on navigation grid and scene query system.

[0089] In the model training using the related technology 1, the scene query system queries a plurality of pathfinding target points, and returns the pathfinding target points meeting the conditions according to the weights, and then moves to the pathfinding target points by means of the navigation grid. Since the moving path of the navigation grid is necessarily reachable, the AI reachability to the pathfinding target points is guaranteed, and the navigation grid movement is a common moving way for the business team to develop various behavior tree AIs.

[0090] The defects of the above related technology 1 include that although the movement from the navigation grid to the pathfinding target points guarantees the AI reachability, this query method only provides a plurality of discrete reachable points for the AI, the navigation grid movement can guarantee the reachability, but the moving path is single and fixed, and cannot completely and clearly indicate all the reachable paths around the AI itself, which limits the diversity of the AI moving path, reduces the completeness of the feasible region, and even if there is a better moving path, the navigation grid movement will miss the better moving path only because the returned pathfinding target point is not on the moving path. In addition, each path query operation based on the pathfinding point will consume the server performance, and the performance consumption is related to the number of pathfinding points and the distance between the pathfinding points and the AI itself. The more the number is and the farther the distance is, the greater the performance consumption is. If this pathfinding scheme is used to train the reinforcement learning AI, when the model prediction frequency is high, the pathfinding point query frequency will also increase, the AI will easily swing between the pathfinding points returned in front and back, and the effect is not good, which will also cause a large amount of performance consumption, and finally the model trained in this way cannot be deployed online.

[0091] Related technology 2, model training method based on atomic action.

[0092] The atomic action defines the smallest instruction operation unit, which is the finest granularity instruction set that cannot be further divided. The atomic action space can completely cover the operations executable by the player, and reaches the level of human players in completeness and flexibility. The moving instruction also adopts the atomic moving way. The atomic moving can completely get rid of the shackles of the navigation grid, and move freely and unconstrained, so as to reach the positions unreachable by the behavior tree and complete the operations that cannot be made by the behavior tree.

[0093] The defects of the above-mentioned related technology 2 include that the granularity of the instruction of atomic action is too fine, in order to fit the operation of a human player, the completeness and fluency of the action are taken into account, so a complex, diverse and high-dimensional atomic action instruction set needs to be designed, and through the coordinated operation of these atomic instruction sets, the operation and movement mode similar to that of a human player can be completed. If an atomic action scheme is used to train a reinforcement learning model, the model may not learn a good strategy or even fail to converge due to the excessively fine atomic action logic and the high dimension of the atomic instruction set, and this influence is particularly prominent when the prediction frequency of the model is high. When the prediction frequency is high, the model will swing between various atomic instructions, and it is easy to appear an inexecutable behavior. Moreover, in the case that the scene map is very large and the terrain is particularly complex, the strategy exploration efficiency of the model is low, and the time cost of training is too high, which finally affects the model effect and production efficiency.

[0094] In combination with the above-mentioned related technology, the related technology has the following disadvantages: first, the scene query combined with the movement mode of the navigation grid is difficult to describe the surrounding road condition information in detail, and the AI movement route is single; second, the design of the pure atomic action instruction set is relatively complex and has a high dimension, and needs to be repeatedly debugged and verified, which has a high time cost; third, in a large map range and a complex and diverse terrain scene, the model trained based on the pure atomic action set has a low learning efficiency, which is not conducive to the model training and convergence; fourth, the strength and personification are difficult to define, and lack quantitative indicators; fifth, the model training method is difficult to apply to a shooting game with a complex and diverse map.

[0095] The embodiment of the present application provides a model training method and device, computer equipment, computer readable storage medium and computer program product, which can improve the efficiency and accuracy of model training. The following describes an exemplary application of the computer equipment provided by the embodiment of the present application. The computer equipment provided by the embodiment of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, and various types of terminals. It can also be implemented as a server. The following describes an exemplary application when the computer equipment is implemented as a server.

[0096] Referring to FIG. 1, FIG. 1 is a network architecture schematic diagram of a model training system 100 provided by the embodiment of the present application. To realize a model training application, a server 400 is connected to a terminal 200 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0097] Taking a first-person shooting game (FPS) as an example, the terminal 200 is configured to collect state data of a first agent in a game of the first-person shooting game, and send the state data to the server 400; the server 400 is configured to use a first prediction model to be trained to predict the state data, obtain a set of action instructions containing a pathfinding mode, control the first agent to perform corresponding actions based on the set of action instructions, obtain execution results, and adjust parameters of the first prediction model to be trained based on reward information of a plurality of execution results obtained in the game, to obtain a trained first prediction model, wherein the reward information is determined based on the plurality of execution results, a main reward coefficient and an auxiliary reward coefficient. The server 400 also returns the trained first prediction model to the terminal 200, and the terminal 200 can use the trained first prediction model to control the first agent to complete the game of the first-person shooting game in an offline manner.

[0098] In the embodiments of the present application, after the server 400 obtains the state data and the first prediction model to be trained, the server 400 uses the first prediction model to be trained to predict the state data of the first agent, and obtains a set of action instructions containing a pathfinding mode. On the one hand, since the first prediction model to be trained can control the first agent to move according to different pathfinding modes based on different scenes in which the first agent is located, when the first agent is in different virtual scenes, a set of action instructions containing different pathfinding modes can be predicted, and the pathfinding modes in the set of action instructions match the scenes in which the first agent is located, thereby achieving diversification of the pathfinding modes. On the other hand, since the trained second prediction model can also control the second agent to move according to different pathfinding modes based on different scenes in which the second agent is located, that is, the second agent can also perform different pathfinding modes in different scenes, thereby strengthening the ability of the first agent to perform different pathfinding modes in different scenes. Then, the first agent is controlled to perform corresponding actions based on the set of action instructions, and execution results are obtained. Finally, reward information of the plurality of execution results is determined using the plurality of execution results, the main reward coefficient and the auxiliary reward coefficient, and parameters of the first prediction model to be trained are adjusted based on the reward information of the plurality of execution results, to obtain the trained first prediction model. In this way, the first agent can perform mixed pathfinding and movement, and the pathfinding mode and the movement mode are diversified, the training effect of the first prediction model to be trained is improved through the diversified pathfinding mode, and the training accuracy of the first prediction model to be trained is also improved since the pathfinding mode of the first agent matches the scene in which the first agent is located. In addition, since the main reward coefficient and the auxiliary reward coefficient are used simultaneously when determining the reward information, the learning efficiency and performance of the first agent in the strategy learning task can be improved, that is, the prediction performance of the first prediction model can be improved.

[0099] In some embodiments, the server 400 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0100] Referring to FIG. 2, FIG. 2 is a structural schematic diagram of the server 400 provided by the embodiments of the present application. The server 400 shown in FIG. 2 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the server 400 are coupled together by a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 440 in FIG. 2.

[0101] The processor 410 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0102] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432 that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0103] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 optionally includes one or more storage devices physically located in proximity to the processor 410.

[0104] The memory 450 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0105] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0106] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0107] The network communication module 452 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), and the like.

[0108] The presentation module 453 is used to enable the presentation of information via one or more output devices 431 associated with the user interface 430 (e.g., a display screen, a speaker, and the like) (e.g., a user interface for operating peripheral devices and displaying content and information).

[0109] The input processing module 454 is used to detect and interpret one or more user inputs or interactions from one or more input devices 432.

[0110] In some embodiments, the device provided by the embodiments of the present application can be implemented in a software manner, and FIG. 2 shows a model training device 455 stored in the memory 450, which can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 4551, an instruction prediction module 4552, an execution module 4553, and an adjustment module 4554. These modules are logical, and thus can be combined or further split according to the functions implemented. The functions of each module will be described below.

[0111] In some embodiments, the device provided by the embodiments of the present application can be implemented in a hardware manner. For example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can be implemented by using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0112] The model training method provided by the embodiments of the present application will be described in combination with an exemplary application and implementation of the server provided by the embodiments of the present application.

[0113] The model training method provided by the embodiments of the present application will be described below. As described above, the computer device implementing the model training method of the embodiments of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated in the following description.

[0114] It should be noted that the examples of the model training method described below are described by taking the first-person shooting game as an example. Those skilled in the art can apply the model training method provided by the embodiments of the present application to model training for other types of games according to the understanding of the following description. The embodiments of the present application can also be applied to various scenarios, including but not limited to games, artificial intelligence, intelligent transportation, and assisted driving.

[0115] Referring to FIG. 3, FIG. 3 is a first flowchart of the model training method provided by the embodiments of the present application. The model training method provided by the embodiments of the present application will be described in combination with the steps shown in FIG. 3, and the execution subject of FIG. 3 is a server.

[0116] In step S101, the state data of the first agent in the virtual scene and the first prediction model to be trained corresponding to the first agent are obtained.

[0117] In the embodiments of the present application, the virtual scene can be a virtual scene in a game match process, and can also be an augmented reality virtual scene or a virtual reality scene. The virtual scene includes the first agent and a second agent corresponding to the trained second prediction model, the second agent being an agent that competes with the first agent, and the trained second prediction model being used to control the second agent to move in different ways based on different scenes in which the second agent is located, and the first prediction model to be trained being used to control the first agent to move in different ways based on different scenes in which the first agent is located.

[0118] In some embodiments, a game match refers to a game confrontation or competition between two or more agents, which can be virtual characters controlled by computer programs, robots or other autonomous entities.

[0119] In the field of artificial intelligence, game matches between agents are commonly used to test and evaluate the performance of different algorithms and methods, and to study the behavior and decision-making ability of agents in complex environments. Such game matches can be conducted in virtual environments or in actual physical environments.

[0120] Game matches between agents can adopt different forms and rules, including but not limited to:

[0121] The first form and rule of the game match is a game of strategy: agents play a game of strategy, such as chess, go, poker, etc., to test their strategy and decision-making ability.

[0122] The second form and rule of the game match is a competitive game: agents play a competitive game, such as a fighting game, a racing game, etc., to test their reaction speed, skill and tactical ability.

[0123] The third form and rule of the game match is a cooperative game: agents play a cooperative game, such as a team cooperation game, a cooperative puzzle game, etc., to test their team cooperation and coordination ability.

[0124] The fourth form and rule of the game match is a mixed game: agents play a variety of forms of game matches, including mixed games of strategy, competition and cooperation, to test their versatility and adaptability.

[0125] Game matches between agents are an important research and practice field in the field of artificial intelligence, through which the behavior and decision-making process of agents can be deeply understood, and the performance and effect of agents can be improved.

[0126] In the embodiments of the present application, the first agent and the second agent are constructed into an adversarial game, that is, in the adversarial game, the first agent and the second agent belong to different camps and are in an antagonistic relationship. Correspondingly, the embodiments of the present application relate to the second game form and rules, that is, the game form and rules adopted in the game in the embodiments of the present application are adversarial games.

[0127] In some embodiments, the state data of the first agent refers to a set of information describing the current state or situation of the first agent. In reinforcement learning and agent design, state data is the basis for the first agent to perceive and understand the environment, and state data contains key features and information of the environment to help the first agent make decisions and actions.

[0128] State data is usually composed of multiple types of information, which can be data about the environment, the first agent itself, and interaction with other entities. State data is the basis for the first agent to interact with the environment, and state data provides the first agent's perception and understanding of the environment, helping the first agent make appropriate decisions and actions based on the current situation. In reinforcement learning, state data is the input of the first agent's policy and value function, directly affecting the learning and decision-making process of the first agent.

[0129] In some embodiments, the first prediction model to be trained corresponding to the first agent refers to a machine learning model used to train the first agent. In reinforcement learning, the first agent is usually represented by a machine learning model, which can be a neural network, decision tree, random forest, etc.

[0130] The first prediction model to be trained refers to a model that needs to be continuously adjusted and optimized according to the interaction data of the first agent and the environment during training. The goal of the first prediction model to be trained is to learn an effective strategy so that the first agent can maximize cumulative rewards or achieve specific task goals when interacting with the environment.

[0131] In reinforcement learning, the first prediction model to be trained usually includes the following main parts:

[0132] Policy Network: used to learn the policy of the first agent, that is, to select the probability distribution of the optimal action given the state.

[0133] Value Network: used to estimate the value of the state or the value of the action, helping the first agent to determine which actions are better or which states are more valuable.

[0134] Action-Value Function: used to estimate the value of taking a specific action in a given state, i.e., the expected cumulative reward of the action.

[0135] Reward Function: used to define the reward signal obtained by the first agent in interacting with the environment, helping the first predictive model to be trained to learn an effective strategy.

[0136] Other auxiliary components: such as experience replay buffer, exploration strategy, etc., used to improve the stability and effectiveness of the model.

[0137] The goal of the first predictive model to be trained is to continuously update and optimize through interaction data with the environment, so that the first agent can gradually learn a better strategy, thereby performing better performance and effectiveness in the actual environment.

[0138] In step S102, the state data is predicted using the first predictive model to be trained to obtain a set of action instructions.

[0139] The set of action instructions includes a pathfinding method and a plurality of action information. In the embodiments of the present application, predicting the state data using the first predictive model to be trained means using the first predictive model to be trained to predict the set of action instructions to be executed by the first agent in the determined state corresponding to the state data. The set of action instructions refers to the set of actions to be executed by the first agent, and the set of action instructions may, for example, include a pathfinding method, a moving direction, a posture type, etc.

[0140] In some embodiments, the first predictive model includes a feature extraction module, a pathfinding prediction module, and a plurality of action prediction modules. Based on this, referring to FIG. 4, the above step S102 can be implemented by the following steps S1021 to S1024, which are described in detail below.

[0141] In step S1021, the state data is feature extracted using the feature extraction module to obtain state features.

[0142] In some embodiments, the state data includes environment information, scalar information, depth map information, and annular ray information, and the environment information, scalar information, depth map information, and annular ray information are information of different modalities. The feature extraction module includes an encoding unit and a feature extraction unit.

[0143] In some embodiments, the environment information refers to various information describing the current game environment, which generally includes various features, attributes and states of the game scene, and can also include various factors affecting the game progress and the decision of the first agent. The scalar information refers to simple and single numerical information describing the game state, which is usually a specific numerical value. Exemplary scalar information can include health points, scores, time, resource quantity, distance, etc. The depth map information refers to image information describing the distance of objects in the game scene from the camera or perspective. The depth map information can provide information about the distance and spatial position of objects in the game world. The ring ray information is usually used in the agent perception system to help the first agent know the situation of the surrounding environment, such as detecting obstacles, obtaining target positions, identifying second agents, etc.

[0144] In some embodiments, referring to FIG. 5, the above step S1021 can be implemented by the following steps S0211 to S0213, which are described in detail as follows.

[0145] In step S0211, the environment information, the scalar information, the depth map information and the ring ray information are respectively encoded by using an encoding unit, so as to obtain encoded environment information, encoded scalar information, encoded depth map information and encoded ring ray information.

[0146] In the embodiments of the present application, the encoding unit can include four encoding sub-units, i.e., a first encoding sub-unit, a second encoding sub-unit, a third encoding sub-unit and a fourth encoding sub-unit.

[0147] In some embodiments, the first encoding sub-unit corresponds to the environment information, and is configured to encode the environment information to obtain the encoded environment information. The first encoding sub-unit can be an embedding model, which maps the environment information into an embedding vector by learning the relationship and pattern of the environment information, so as to convert the environment information into a more compact and efficient representation form. The second encoding sub-unit corresponds to the scalar information, and is configured to encode the scalar information to obtain the encoded scalar information. The second encoding sub-unit can also be an embedding model similar to the first encoding sub-unit. The third encoding sub-unit corresponds to the depth map information, and is configured to encode the depth map information to obtain the encoded depth map information. The third encoding sub-unit can be a two-dimensional convolution (Conv2D) model. The fourth encoding sub-unit corresponds to the ring ray information, and is configured to encode the ring ray information to obtain the encoded ring ray information. The fourth encoding sub-unit can be a one-dimensional convolution (Conv1D) model.

[0148] In step S0212, the encoded environment information, the encoded scalar information, the encoded depth map information and the encoded annular ray information are spliced to obtain comprehensive encoded information.

[0149] In the embodiments of the present application, splicing the encoded environment information, the encoded scalar information, the encoded depth map information and the encoded annular ray information means combining the encoded environment information, the encoded scalar information, the encoded depth map information and the encoded annular ray information to obtain comprehensive encoded information.

[0150] The embodiments of the present application do not limit the combination order when combining. For example, the combination can be performed according to the arrangement order of the encoded environment information, the encoded scalar information, the encoded depth map information and the encoded annular ray information, or the combination can be performed according to the arrangement order of the encoded scalar information, the encoded environment information, the encoded depth map information and the encoded annular ray information.

[0151] In step S0213, the feature extraction unit is used to perform feature extraction on the comprehensive encoded information to obtain state features.

[0152] In the embodiments of the present application, the feature extraction unit can be a deep neural network. For example, the feature extraction unit can be a long short-term memory artificial neural network. The feature extraction unit encapsulates and fuses the comprehensive encoded information to obtain state features. The feature extraction unit can capture long-term dependencies in the comprehensive encoded information.

[0153] Through the above steps S0211 to S0213, the information of different modalities included in the state data is respectively encoded. In this way, the features of each modality can be better captured, thereby improving the accuracy of data processing and analysis, and the details of each modality can also be retained, so that the data is more comprehensive and rich in the transmission and analysis process. In addition, the feature extraction unit fuses different modal information, which can enhance the interaction and information integration between modalities. Through feature extraction, state features with rich information are obtained. Therefore, the state features can accurately reflect the situation of the game, so as to provide accurate data basis for model training, thereby improving the accuracy of the trained first prediction model.

[0154] Still referring to FIG. 4, the step S1021 is continued.

[0155] In step S1022, for each action prediction module, the action prediction module is used to perform prediction based on the state features to obtain action information of the first agent.

[0156] In the embodiments of the present application, the action prediction module corresponds to other actions that can be performed by the first agent, and the other actions refer to actions other than the pathfinding action. Since the pathfinding action and the other actions are described separately in the embodiments of the present application, the number of action prediction modules is one less than the number of actions that can be performed by the first agent. For example, assuming that the first agent can perform 9 actions, there are 8 action prediction modules.

[0157] In some embodiments, the action prediction module is configured to perform action prediction based on the state feature, so as to obtain action information of the first agent, and the action information is used to instruct the first agent to perform the action.

[0158] Referring to FIG. 6, the above step S1022 can be implemented by the following steps S0221 to S0225, which are described in detail as follows.

[0159] In step S0221, the second action identifier corresponding to the action prediction module is obtained.

[0160] In the embodiments of the present application, each action has a corresponding action identifier, and therefore, the action prediction module also has a second action identifier of the predicted action.

[0161] In step S0222, based on the action sequence table, the third action identifier located in front of the second action identifier is determined.

[0162] In the embodiments of the present application, the action sequence table is used to represent the arrangement order of the actions, and the action sequence table is a pre-set table. The action sequence table stores the action identifiers of each action in sequence, and the action sequence table can represent the sequence of the actions. The sequence of the actions can reflect the dependency relationship between the actions. For example, the action identifiers stored in the action sequence table are 01, 02, 03, 04, 05, 06, 07, 08, and 09, wherein 01 is the action identifier corresponding to the horizontal aiming direction of the gun barrel, 02 is the action identifier corresponding to the vertical aiming direction of the gun barrel, 03 is the action identifier corresponding to whether to fire, 04 is the action identifier corresponding to the moving mode, 05 is the action identifier corresponding to the pathfinding mode, 06 is the action identifier corresponding to the moving direction, 07 is the action identifier corresponding to the posture type, 08 is the action identifier corresponding to the side type, and 09 is the action identifier corresponding to the special type. Therefore, the arrangement order of the actions is the horizontal aiming direction of the gun barrel, the vertical aiming direction of the gun barrel, whether to fire, the moving mode, the pathfinding mode, the moving direction, the posture type, the side type, and the special type.

[0163] In some embodiments, the third action identifier refers to the action identifier located before the second action identifier and adjacent to the second action identifier. Based on the above example, assuming that the second action identifier is 03, the third action identifier is 02.

[0164] In some embodiments, before performing the above step S0222, a preset action sequence table can also be acquired first.

[0165] In step S0223, second action information corresponding to the third action identifier is acquired.

[0166] In the embodiments of the present application, the second action information is predicted by using the action prediction module corresponding to the third action identifier.

[0167] Continuing the above example, the third action identifier is 02, and the second action information is predicted by using the action prediction module corresponding to 02.

[0168] In step S0224, the second action information is encoded to obtain second encoded action information.

[0169] In the embodiments of the present application, the encoding of the second action information can be realized by embedding processing of the second action information, and the obtained embedding vector is determined as the second encoded action information.

[0170] In step S0225, the action prediction module corresponding to the second action identifier is used to predict based on the state feature and the second encoded action information to obtain the action information of the first agent.

[0171] In the embodiments of the present application, the second action is not only predicted based on the state feature, but also based on the second encoded action information of the previous action, so as to obtain the action information corresponding to the second action.

[0172] Through the above steps S0221 to S0225, when the action prediction module is used to predict the action information of the second action, the state information and the second encoded action information of the previous action are used as the basis, so that the action prediction module can better understand the context information of the action, the dependency relationship between actions is integrated when predicting, the continuity of the previous action and the second action can be ensured, and an accurate prediction result, i.e., accurate action information, is obtained, so as to provide an accurate data basis for model training, thereby improving the accuracy of the trained first prediction model.

[0173] In some embodiments, the action information of each action except the routing action can be determined based on the above steps S0221 to S0225.

[0174] Continuing to refer to FIG. 4, the step S1022 is explained.

[0175] In step S1023, the routing prediction module is used to predict based on the state feature to obtain the routing mode of the first agent.

[0176] In the embodiments of the present application, the pathfinding prediction module is used to predict the pathfinding manner.

[0177] Referring to FIG. 7, the above step S1023 can be implemented by the following steps S0231 to S0235, which are explained in detail as follows.

[0178] In step S0231, a preset action sequence table is obtained.

[0179] In the embodiments of the present application, the action sequence table is used to represent the arrangement sequence between actions. In some embodiments, the action sequence table can be obtained by obtaining an instruction.

[0180] In step S0232, based on the action sequence table, a first action identifier located in the front sequence of the pathfinding action identifier is determined.

[0181] In some embodiments, the implementation process of the above step S0232 is similar to that of the above step S0222, and thus the implementation process of the above step S0232 can refer to that of the above step S0222. The first action identifier located in the front sequence of the pathfinding action identifier refers to an action identifier located before the pathfinding action identifier and adjacent to the pathfinding action identifier. Taking the above example, the pathfinding action identifier is 05, and thus the first action identifier is 04.

[0182] In step S0233, first action information corresponding to the first action identifier is obtained.

[0183] In the embodiments of the present application, the first action information is predicted by using the action prediction module corresponding to the first action identifier.

[0184] In some embodiments, the implementation process of the above step S0233 is similar to that of the above step S0223, and thus the implementation process of the above step S0233 can refer to that of the above step S0223.

[0185] In step S0234, the first action information is encoded to obtain first encoded action information.

[0186] In some embodiments, the implementation process of the above step S0234 is similar to that of the above step S0224, and thus the implementation process of the above step S0234 can refer to that of the above step S0224.

[0187] In step S0235, the pathfinding prediction module is used to predict based on the state feature and the first encoded action information to obtain the pathfinding manner of the first agent.

[0188] In the embodiment of the present application, when predicting the pathfinding manner, not only the state feature is used as the basis, but also the first encoded action information, that is, the encoded action information of the previous action, is used as the basis. In this way, the pathfinding prediction module can predict the influence of the subsequent action on path planning based on the analysis of the previous action, thereby optimizing the pathfinding strategy, so that a more efficient or state feature conforming pathfinding manner is obtained, and the accuracy of the pathfinding manner is improved.

[0189] Referring to FIG. 8, the above step S0235 can be implemented by the following steps S2351 to S2353, which are specifically described below.

[0190] In step S2351, the state feature and the first encoded action information are predicted by using the pathfinding prediction module according to a prediction period, to obtain a target pathfinding index.

[0191] In the embodiment of the present application, the target pathfinding index is one of a first pathfinding index and a second pathfinding index, wherein the first pathfinding index and the second pathfinding index are different pathfinding indexes, and the first pathfinding index and the second pathfinding index represent different pathfinding manners. For example, the first pathfinding index represents a pathfinding manner based on atomic movement, and the second pathfinding index represents a pathfinding manner according to a rule movement.

[0192] In some embodiments, the prediction period is a value set in advance. For example, the value of the prediction period can be 6 seconds, 7 seconds, or 8 seconds.

[0193] In some embodiments, the pathfinding prediction module predicts the target pathfinding index, and the target pathfinding index can indicate a target pathfinding manner.

[0194] In step S2352, a target pathfinding manner corresponding to the target pathfinding index is determined based on a preset index mapping table.

[0195] In some embodiments, the index mapping table stores a mapping relationship between a pathfinding index and a pathfinding manner. Based on this, the target pathfinding manner corresponding to the target pathfinding index can be determined from the index mapping table.

[0196] In step S2353, the target pathfinding manner is determined as the pathfinding manner of the first agent.

[0197] In some embodiments, when the target pathfinding index is the second pathfinding index, it is determined that the pathfinding manner of the first agent is a pathfinding manner according to a rule, which can also be referred to as a pathfinding manner according to a rule movement. When the target pathfinding index is the first pathfinding index, it is determined that the pathfinding manner of the first agent is a pathfinding manner based on atomic movement.

[0198] The rule-based pathfinding manner refers to a path planning strategy that relies on predefined rules or instructions to guide a mobile entity (such as a robot, an autonomous vehicle, etc.) to move from a starting position to a target position in an environment. The predefined rules can be obstacle avoidance rules, priority rules, greedy search rules, etc. The atomic movement-based pathfinding manner is a path planning strategy at a microscopic level, which regards the mobile entity (such as a robot, a vehicle, etc.) as an "atom" that can move freely, and calculates the path according to its physical characteristics or dynamics model.

[0199] By the above steps S0231 to S0235, when the pathfinding action is predicted by the pathfinding prediction module, the dependence between the pathfinding action and the first action is decoupled based on the state information and the first encoded action information, so that the dependence between the pathfinding action and the first action is integrated when the prediction is performed, thereby obtaining a pathfinding manner with high accuracy, providing training data with higher accuracy for model training. Not only can it speed up the convergence speed, reduce the training time and iteration number, and improve the training efficiency, but also the training data with higher accuracy can reduce the risk of learning noise and irrelevant features by the model, thereby avoiding overfitting phenomenon. In addition, the model is trained on training data with higher accuracy, which can better identify and handle abnormal situations, enhance the robustness of the model, and thus improve the prediction accuracy and generalization ability of the model.

[0200] Continuing to refer to FIG. 4, the step S1023 is explained.

[0201] In step S1024, a set of action instructions is constructed based on the pathfinding manner and the action information of each action.

[0202] In the embodiment of the present application, the set of action instructions includes the action information of all actions, and the set of action instructions is used to instruct the first agent to perform the corresponding action.

[0203] Referring to FIG. 9, the above step S1024 can be implemented by the following steps S0241 to S0243, which are specifically explained as follows.

[0204] In step S0241, a preset action mask is obtained.

[0205] In the embodiment of the present application, the action mask matches the interaction rule in the virtual scene, and the action mask is used to limit the action selection space of the first agent. When the virtual scene is a virtual scene in a game match process, the interaction rule can be a game rule.

[0206] In some embodiments, the action mask can be a mechanism to limit the range of actions that the first agent can perform in a given state, helping the first agent to make more appropriate decisions in a specific situation. The action mask can be considered as a binary vector, where each element corresponds to an action. When an action mask value is 1, it means that the action is executable; while when the mask value is 0, it means that the action is not executable.

[0207] In some embodiments, it is first necessary to clarify the interaction rules, which include which actions can be executed and which actions are prohibited in a specific situation. For example, in a chess game, the game rules can prohibit the king from moving to a position that will be attacked by the opponent. Then according to the game rules, an action space is created, which contains all possible actions that the agent can take. In the game session, according to the current game state, it is determined which actions are allowed to be executed and which actions are not allowed to be executed, and then the mask value corresponding to the allowed actions is set to 1 and the mask value corresponding to the disallowed actions is set to 0, thereby obtaining the action mask.

[0208] In step S0242, the pathfinding mode and the action information are combined to form a candidate instruction set.

[0209] In the embodiments of the present application, the pathfinding mode and the action information are combined to form a candidate instruction set.

[0210] In step S0243, the candidate instruction set is masked using the action mask to obtain an action instruction set.

[0211] In the embodiments of the present application, the action information includes a first flag value for indicating whether to execute an action. The first flag value can be a first preset value or a second preset value. When the first flag value is the first preset value, it indicates that the action corresponding to the action information is not executed, and when the first flag value is the second preset value, it indicates that the action corresponding to the action information is executed. Exemplarily, the first preset value can be 0 and the second preset value can be 1. The action mask can be a one-dimensional vector, each vector value being a mask value, and each mask value corresponding to an action information. The mask value is the first preset value or the second preset value. When the mask value is the first preset value, it means that the action information corresponding to the mask value needs to be masked, and when the mask value is the second preset value, it means that the action information corresponding to the mask value does not need to be masked.

[0212] When the candidate instruction set is masked by the action mask, for each action information in the candidate instruction set, a mask value corresponding to the action information is obtained from the action mask; when the mask value corresponding to the action information is a first preset value, and a first flag value in the action information is a second preset value, the first flag value in the action information is updated to the first preset value. That is, when the mask value corresponding to an action information is 0, it means that the action corresponding to the action information needs to be masked, that is, the action corresponding to the action information is not executed, and then it is needed to continue to determine whether the first flag value in the action information of the action is 1. If the first flag value is 1, it represents that the action is executed, and then the first flag value in the action information of the action is updated to 0, so that the action is not executed. If the first flag value in the action information is not 1, that is, 0, it means that the action information itself represents that the action is not executed, and at this time the action information is kept unchanged.

[0213] In some embodiments, when the mask value corresponding to an action information is a second preset value, for example, the mask value is 1, it means that the action information is not masked, and at this time the action information can be kept unchanged.

[0214] In this way, the action instruction set is obtained by filtering the candidate instruction set by the action mask.

[0215] Through the above steps S0241 to S0243, the action instruction set is obtained by filtering and screening the pathfinding mode and each action information by the action mask. The action space is constrained, so that the action instruction is more consistent with the actual situation or task requirement; and the behavior strategy is optimized, so that the decision efficiency and performance of the first intelligent agent are improved, the length of model training is shortened, and the efficiency and accuracy of model training are improved.

[0216] Through the above steps S1021 to S1024, different modal information is encoded respectively, so that the situation state of the game match can be accurately represented. In addition, the feature extraction unit fuses different modal information, and obtains state features with rich information quantity through feature extraction. Then, the dependency relationship between actions is integrated when predicting, so that the prediction result with high accuracy is obtained. Finally, the pathfinding mode and each action information are filtered and screened by the action mask, the action space is constrained, and the behavior strategy is optimized, so that the decision efficiency and performance of the first intelligent agent are improved, and the efficiency and accuracy of model training are improved.

[0217] Continue to refer to FIG. 3, and the step S102 is explained as follows.

[0218] In step S103, the first agent is controlled to move based on the pathfinding manner, and perform corresponding actions based on each of the action information, to obtain an execution result.

[0219] In the embodiments of the present application, each action in the set of action instructions is executed by the first agent to change the environment of the game session, so as to obtain an execution result.

[0220] In some embodiments, when the pathfinding manner of the first agent is the rule-based pathfinding manner, referring to FIG. 10, the step S103 of “controlling the first agent to move based on the pathfinding manner, and performing corresponding actions based on each of the action information” can be implemented by the following steps S1031 to S1036, which are described in detail as follows.

[0221] In step S1031, a target position and a current position of the first agent are obtained.

[0222] In some embodiments, the target position can be obtained in an advance preset manner. For example, the target position can be a fixed position or a specific object position. The target position can also be a randomly generated position, so as to increase the randomness and challenge of the game. The target position can also be a position obtained based on environment information or a task setting.

[0223] In some embodiments, the current position refers to the current position of the first agent.

[0224] In step S1032, a moving path of the first agent is determined based on the target position and the current position.

[0225] In some embodiments, the moving path can be determined based on the target position and the current position by using a path planning algorithm, where the path planning algorithm can be an ant colony algorithm, a Dijkstra algorithm, etc.

[0226] Herein, taking the Dijkstra algorithm as an example of the path planning algorithm, the implementation process of determining the moving path of the first agent is described. First, the map data of the virtual scene in which the first agent is located is acquired, and the map data includes a plurality of nodes. Then, the distance values of all nodes are set to infinity (indicating that the distance from the starting node to the node is unknown). The node corresponding to the current position is determined as the starting node, and the distance value of the starting node is set to 0 (indicating that the distance from the starting node to itself is 0). Then, a priority queue is created, and all nodes are added to the priority queue, and the starting node has the highest priority (the smallest distance). The node closest to the starting node in the priority queue is taken out and determined as the current node, and then all unvisited adjacent nodes of the current node are traversed. For each adjacent node, the distance from the starting node to the node is calculated. If the calculated distance is smaller than the current recorded distance of the adjacent node, the distance of the adjacent node is updated, and the node is put into the priority queue. The current node is marked as visited to prevent it from being processed again. The process of selecting the nearest unvisited node, updating the distance of the adjacent node, and marking the current node as visited is repeated until the target node is marked as visited or the priority queue is empty (all nodes have been visited). If the target node corresponding to the target position is visited, the path planning result, i.e., the moving path of the first agent, is obtained by backtracking from the target node to the starting node according to the recorded predecessor node of each node (the node pointing to it in the shortest path tree).

[0227] In step S1033, based on the moving path, the first agent is controlled to move in a regular pathfinding manner, and based on the action information, the first agent is controlled to perform each action.

[0228] In some embodiments, the first agent is controlled to move in the moving path, and based on the action information, the first agent is controlled to perform each action.

[0229] In step S1034, during the process of controlling the first agent to move in the regular pathfinding manner, when it is determined that the pathfinding manner switching condition is met, the target pathfinding index is updated to the first pathfinding index.

[0230] In some embodiments, during the process of controlling the first agent to move in the regular pathfinding manner, after the moving path is determined, the target pathfinding index can be updated to the third pathfinding index, and the third pathfinding index is used to represent that the pathfinding manner remains unchanged or the moving manner remains unchanged, i.e., the moving path remains unchanged, and the first agent moves based on the moving path in a regular manner, without the need to determine the moving path again.

[0231] In some embodiments, when it is determined that the pathfinding mode switching condition is met, the target pathfinding index can be first updated to a fourth pathfinding index, the fourth pathfinding index being used to represent a suspension of the rule-based pathfinding mode, so as to timely suspend the current pathfinding mode; then, the target pathfinding index is updated from the fourth pathfinding index to the first pathfinding index.

[0232] In some embodiments, based on the received game session data, the implementation manner of determining that the pathfinding mode switching condition is met includes the following three implementation manners:

[0233] Implementation manner one, when it is determined that the first agent moves to the target position, it is determined that the pathfinding mode switching condition is met.

[0234] In this implementation manner, the representation that the first agent has reached the target position also represents that the first agent has completed the traversal of the movement path, and at this time it is determined that the pathfinding switching condition is met.

[0235] Implementation manner two, when it is determined that the first agent is switched from an occlusion scene to a non-occlusion scene, it is determined that the pathfinding mode switching condition is met, wherein the occlusion scene represents that there is an occlusion between the first agent and the second agent, and the non-occlusion scene represents that there is no occlusion between the first agent and the second agent.

[0236] In this implementation manner, the first position coordinate of the first agent and the second position coordinate of the second agent can be first obtained, based on the first position coordinate and the second position coordinate, a straight line between the first agent and the second agent is determined, and a ray detection is performed along the straight line between the first agent and the second agent to determine whether the ray intersects with an object in the game scene. If the ray intersects with the object in the game scene, it means that there is an occlusion between the first agent and the second agent; if the ray does not intersect with any object in the game scene, it means that there is no occlusion between the first agent and the second agent. There is no occlusion between the first agent and the second agent, which means that the first agent and the second agent can see each other, and also represents that the game session enters from a non-battle scene to a battle scene, at this time it is determined that the game session data meets the pathfinding switching condition, thereby providing path protection for the battle between the two parties, and the pathfinding mode of atomic movement enables the first agent to quickly respond to the battle scene.

[0237] Implementation manner three, a movement duration of the first agent moving in the rule-based pathfinding mode is obtained, and when the movement duration reaches a duration threshold, it is determined that the pathfinding mode switching condition is met, wherein the duration threshold is less than a prediction period.

[0238] In this implementation, the target position and the movement path do not change or update during the movement duration, and in order to adapt to the variability and real-time performance of the game, it is determined that the game session data meets the pathfinding switching condition, and the target pathfinding index is updated to the first pathfinding index.

[0239] In some embodiments, the duration threshold is a value set in advance. For example, the duration threshold can be 3 seconds or 4 seconds.

[0240] In step S1035, the pathfinding mode of the first agent is switched to the pathfinding mode according to atomic movement.

[0241] In the embodiments of the present application, the first pathfinding index corresponds to the pathfinding mode according to atomic movement, and therefore, the pathfinding mode of the first agent is switched to the pathfinding mode according to atomic movement.

[0242] In step S1036, the first agent is controlled to perform atomic movement.

[0243] In some embodiments, the first agent is controlled to perform atomic movement for the remaining time of the prediction period until the prediction period ends.

[0244] Through the above steps S1031 to S1036, when the target pathfinding index is the second pathfinding index, the movement path is first determined, and then the first agent is controlled to move according to the regular pathfinding mode based on the movement path. In this way, the movement path only needs to be determined once, thereby saving computing resources, shortening the model training duration, and improving the model training efficiency. Then, during the movement of the first agent according to the regular pathfinding mode, if it is determined that the pathfinding mode switching condition is met, the target pathfinding index is updated to the first pathfinding index, and the first agent is controlled to perform atomic movement until the prediction period ends. In this way, when the pathfinding mode switching condition is met, the pathfinding mode is switched in time, which not only improves the flexibility of the pathfinding mode, but also improves the adaptability of the pathfinding mode to the scene in which the first agent is located, thereby providing accurate training data for model training, and further improving the prediction accuracy of the model.

[0245] In some embodiments, when the pathfinding mode of the first agent is the pathfinding mode according to atomic movement, the first agent is controlled to perform atomic movement throughout the prediction period.

[0246] Continuing to refer to FIG. 3, the step S103 is explained in continuation.

[0247] In step S104, the parameters of the first prediction model to be trained are adjusted based on the reward information of the plurality of execution results, and a trained first prediction model is obtained.

[0248] In the embodiments of the present application, the reward information is determined based on the multiple execution results, the main reward coefficient and the auxiliary reward coefficient.

[0249] In some embodiments, the parameters of the first prediction model to be trained can be adjusted based on the reward information of each execution result respectively, or the parameters of the first prediction model to be trained can be adjusted based on the sum of the reward information of each execution result.

[0250] Referring to FIG. 11, the above step S104 can be implemented by the following steps S1041 to S1043, which are specifically described as follows.

[0251] In step S1041, the reward function is determined based on the main reward coefficient and the auxiliary reward coefficient.

[0252] In the embodiments of the present application, the main reward coefficient is used to represent the reward information given when the first agent performs the action corresponding to the action information in the set of action instructions, and the auxiliary reward coefficient is used to represent the reward coefficient given when the first agent performs the preset humanization action. The main reward coefficient and the auxiliary reward coefficient can be preset.

[0253] In some embodiments, the preset humanization action can be a proactive search for cover action, a fire pulling action, a distance pulling action, etc.

[0254] In some embodiments, the reward function can be determined by the following formula (1): R = a · R main + b · R auxiliayr formula (1) ;

[0255] In formula (1), R represents the reward information, a represents the main reward coefficient, b represents the auxiliary reward coefficient, R main represents the main reward value, and R auxiliayr represents the auxiliary reward value.

[0256] The main reward value is usually closely related to the goal of completing the main task of the first agent. For example, if the main task of the agent is to reach a certain destination, the main reward value obtained after reaching the destination will be relatively high. The main reward value can be set according to the task completion degree, efficiency or other related indicators. The auxiliary reward value is the reward brought by the secondary goal or behavior related to the main task. For example, avoiding enemies, maintaining a healthy state, saving resources, etc. The auxiliary reward value can help the agent form a more comprehensive and stable strategy in the learning process.

[0257] In step S1042, the reward information of each execution result is determined by using the reward function.

[0258] In some embodiments, the execution result refers to a state or feedback obtained by the first agent after performing the movement based on the pathfinding mode and the corresponding action. The execution result can include at least one of the current position of the first agent, the interaction result (such as collecting an item, attacking an enemy, etc.) of the first agent with an object or element in the virtual environment, the change of the environment after the agent performs the action (such as the state of the enemy, the state of the object, etc.), the game score after the agent performs the action. In implementation, step 1042 first determines the main reward value and the auxiliary reward value based on the execution result, and then substitutes the main reward value and the auxiliary reward value corresponding to the execution result into formula (1) to obtain the reward information of the execution result. The reward information can be the total reward value.

[0259] In step S1043, the reward information of each execution result is accumulated to obtain a cumulative reward, the cumulative reward is back-propagated to the first prediction model to be trained, and the parameters of the first prediction model to be trained are adjusted based on the cumulative reward until a training end condition is reached to obtain the trained first prediction model.

[0260] In some embodiments, the reward information of the execution result in one or more prediction cycles can be accumulated to obtain a cumulative reward, and then the parameters of the first prediction model to be trained are adjusted based on the cumulative reward. When a preset adjustment number is reached or the cumulative reward reaches a reward threshold, it is determined that the training end condition is reached, thereby obtaining the trained first prediction model.

[0261] Through the above steps S1041 to S1043, by introducing the auxiliary reward coefficient, the first prediction model to be trained can be trained towards the direction of the preset humanized action, so that the trained first prediction model can control the first agent to have the humanized style while ensuring the ability strength, thereby improving the prediction performance of the trained first prediction model.

[0262] In some embodiments, after obtaining the trained first prediction model, after the start of the game match, the trained first prediction model can be used to predict based on the state data of the first agent to obtain a set of action instructions, and then control the first agent to move based on the set of action instructions and perform the corresponding action based on each action information in the set of action instructions, thereby interacting with a virtual object. The virtual object can be a game character that competes with the first agent. The virtual object can be a game character controlled by a real player, or a second agent driven by a second prediction model.

[0263] Through the steps S101 to S104, since the first prediction model to be trained can control the first agent to move in different pathfinding manners based on different scenes in which the first agent is located, when the first agent is in different scenes in the game match, a set of action instructions containing different pathfinding manners can be predicted, and the pathfinding manners in the set of action instructions match the scenes in which the first agent is located, thereby achieving diversification of the pathfinding manners. On the other hand, since the trained second prediction model can also control the second agent to move in different pathfinding manners based on different scenes in which the second agent is located, that is, the second agent can also perform different pathfinding manners in different scenes, thereby strengthening the ability of the first agent to perform different pathfinding manners in different scenes. The pathfinding manner and the moving manner are diversified, the training effect of the first prediction model to be trained is improved through the diversified pathfinding manners, and the training accuracy of the first prediction model to be trained can be improved. In addition, through the auxiliary reward coefficient, the first prediction model to be trained can be trained in the direction of the preset humanized action.

[0264] In some embodiments, after receiving the game match instruction, the first initial position of the first agent and the second initial position of the second agent are first determined, and the first agent is generated at the first initial position and the second agent is generated at the second initial position. Based on this, referring to FIG. 12, the following steps S001 to S009 can also be executed before the above step S101, which will be specifically described below.

[0265] In step S001, in response to a game match start instruction, a selected region is determined from each sub-region covered by a virtual scene corresponding to the game match based on the equal probability principle.

[0266] In some embodiments, the game match start instruction can be in the form of touch, gesture, voice, etc.

[0267] In some embodiments, the probability weight of each sub-region can be set to the same value, and each sub-region is associated with a number, and then a random number generator is used to generate a random number, and the sub-region associated with the random number is determined as the selected region.

[0268] In step S002, the first center point of the selected region is determined.

[0269] In some embodiments, the coordinates of the first center point of the selected region can be determined by using a geometric center method, a barycenter method, a point set method, a distance weighting method, etc.

[0270] In step S003, based on the first center point and a preset first maximum pathfinding distance, a first birth region is determined, and a first set of available points in the first birth region is determined.

[0271] In some embodiments, a first center point can be taken as the center of a circle, and a first maximum pathfinding distance can be taken as the radius to determine a circular first birth area. Alternatively, the first center point can be taken as the center, and the first maximum pathfinding distance can be taken as the side length to determine a square first birth area.

[0272] In some embodiments, the first navigable points refer to valid positions or areas where the first agent can move. These points are usually considered as target points when the first agent plans a path and navigates in the game session. The determination of the first navigable points is usually based on the following aspects:

[0273] Firstly, terrain features: the navigable points are usually located in flat and passable terrain areas in the game map, such as flat ground, roads, stairs, etc. These areas are usually safe for the first agent to move and operate.

[0274] Secondly, obstacle avoidance: the navigable points usually need to avoid obstacles or obstacle areas in the game map, such as walls, buildings, water areas, etc. The first agent needs to consider how to bypass these obstacles when planning a path to find a safe path.

[0275] Thirdly, target points: the navigable points usually also include target points or target areas in the game, such as task targets, enemy positions, resource points, etc. The first agent can find the best path to reach these target points through path planning.

[0276] Fourthly, player interaction points: the navigable points can also include points or areas where the player can interact, such as shops, NPC locations, transportation tools, etc. The first agent can choose appropriate paths and target points according to the player's needs and behaviors.

[0277] By determining the navigable points in the game map, the first agent can be provided with effective movement targets and path planning references, so that it can exhibit more intelligent and flexible behaviors in the game.

[0278] In the embodiments of the present application, all the first navigable points constitute a first navigable point set.

[0279] In step S004, a first initial position of the first agent is determined from the first navigable point set, and the first agent is generated at the first initial position.

[0280] In some embodiments, any first navigable point in the first navigable point set can be randomly determined as the first initial position, and the first agent is generated at the first initial position, i.e., the first initial position is the birth position of the first agent.

[0281] In step S005, the first initial position is determined as the second center point.

[0282] In step S006, a preset second maximum path searching distance corresponding to the second agent and a preset second minimum path searching distance corresponding to the second agent are obtained.

[0283] In some embodiments, the preset second maximum path searching distance and the preset second minimum path searching distance can be preset values in advance.

[0284] In step S007, based on the second center point, the preset second maximum path searching distance and the preset second minimum path searching distance, a second birth area is determined, and a second available point set in the second birth area is determined.

[0285] In some embodiments, the second center point can be taken as the center of a circle, the preset second maximum path searching distance can be taken as the outer radius of the circle, and the preset second minimum path searching distance can be taken as the inner radius of the circle, so as to determine a circular ring, and the circular ring is determined as the second birth area.

[0286] In some embodiments, the implementation process of determining the second available point set in the second birth area is similar to the implementation process of “determining the first available point set in the first birth area” in step S003 described above, and therefore, the implementation process of determining the second available point set in the second birth area can refer to the implementation process of “determining the first available point set in the first birth area” in step S003 described above.

[0287] In step S008, a candidate initial position is determined from the second available point set.

[0288] In some embodiments, in some embodiments, any second available point in the second available point set can be randomly determined as the candidate initial position.

[0289] In step S009, when there is an obstacle between the candidate initial position and the first initial position, the candidate initial position is determined as the second initial position of the second agent, and the second agent is generated at the second initial position.

[0290] In the embodiments of the present application, when there is an obstacle between the candidate initial position and the first initial position, the candidate initial position is determined as the second initial position of the second agent, and the second agent is generated at the second initial position.

[0291] In some embodiments, when there is no obstacle between the candidate initial position and the first initial position, the above step S008 is returned to, and the candidate initial position is determined again.

[0292] Through the steps S001 to S009, on the one hand, the randomness generated by the first agent and the second agent can be improved, the probability of each region as a selected region is improved, and the complete symmetry of the first agent and the second agent in the selected region is ensured; on the other hand, through the existence of obstacles between the two birth positions, the diversity of the sample is ensured, the situation that one party is killed by the other party at the beginning of the game is avoided, the game time is improved, sufficient training samples are ensured, and the model training efficiency is improved.

[0293] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0294] The embodiments of the present application provide a model training method. The model training method is based on a large-scale reinforcement learning training framework, combined with a proximal policy optimization algorithm, and in a super-large map scenario, a complex and diverse three-dimensional terrain, and a high-dimensional structured action space, a rule-enhanced hybrid movement pathfinding scheme is proposed in a manner of combining atomic movement and navigation grid movement. Then, the powerful adaptive learning and exploration capability of a neural network is used to realize a PVE and PVP companion AI training scheme that can be deployed in a super-large map and a complex and diverse terrain, wherein the companion AI corresponds to the first agent in other embodiments.

[0295] The embodiments of the present application additionally introduce navigation grid movement as a global movement logic on the basis of the original action space, and adopt a centralized training and hybrid execution model architecture. In a battle scenario, the AI focuses on learning a delicate positioning. At this time, the neural network focuses more on selecting an atomic movement mode, so as to learn the most exquisite micro-operation and make an operation similar to that of a human player, to improve the human-likeness of the AI. In a non-battle scenario, the neural network will weaken the pathfinding learning process and tend to select a global movement mode of the navigation grid, so as to be able to quickly encounter a target enemy and quickly switch to a battle scenario, so that more battle samples can be produced in a game, and the training efficiency is improved, wherein the target enemy corresponds to the second agent in other embodiments.

[0296] In some embodiments, a complete auxiliary reward system is also introduced to enhance the human-likeness of the strategy. In a two-person zero-sum game scenario, self-game training is used to continuously improve the strength of the strategy. The hybrid pathfinding reinforcement learning training scheme can fully utilize the decision-making and exploration advantages of reinforcement learning, and adaptively and flexibly select a pathfinding mode according to the game situation. Not only can the global pathfinding movement and the battle game ability be taken into account, but also the training efficiency can be improved, and a high-level companion AI that meets the business requirements can be produced in a super-large map and a complex and diverse three-dimensional terrain scenario, to help the business team to more efficiently generate content.

[0297] Through the model training method provided by the embodiments of the present application, the following four purposes can be achieved:

[0298] The first purpose is to accelerate the collection of training data for policy learning through large-scale distributed reinforcement learning; the second purpose is to avoid the problem that the accompanying AI character is trapped in a certain position or cannot move as expected when moving or performing actions due to the terrain design, through a hybrid pathfinding method enhanced by rules, in cooperation with centralized training of reinforcement learning; the third purpose is to train an accompanying AI that takes into account full-map navigation and battle gameplay in a super-large map range and complex and variable three-dimensional terrain scenes, not only to improve the upper limit of model effectiveness, but also to greatly accelerate training efficiency; the fourth purpose is to enhance the model's humanization and style guidance through a comprehensive reward and punishment mechanism, further accelerating model adaptation and landing. The model corresponds to the first prediction model to be trained in other embodiments.

[0299] Next, the model training method provided by the embodiments of the present application will be described from three aspects. The first aspect is the business target, which is the task background on which the embodiments of the present application rely, and needs to clarify the tasks and specific roles undertaken by AI. The second aspect is environmental interaction, which has universality, and the network game using state synchronization mode can use this set of training and reasoning mode. The third aspect is the training scheme, and the embodiments of the present application propose a differentiated training scheme. The introduction of the hybrid pathfinding method accelerates the training efficiency and improves the upper limit of the model. The highly integrated neural network and the complete reward and punishment system are beneficial to guiding the model to humanize the style and help to improve the model's humanization.

[0300] Firstly, the business target.

[0301] Since the farm map is the map with the highest player activity, taking the farm map as an example, the related technology has already put the behavior tree version of ordinary AI for accompanying play and filling the number of games, but since the existing behavior tree version of AI logic is easy to be recognized by players, it leads to the loss of fun of players in the battle. Based on this, the business target is to produce high-intensity and highly humanized AI through reinforcement learning technology for accompanying play in high-end games, which requires full-map movement in the whole map, reasonable use of cover battle and pulling, flexible and natural walking, human-like shooting performance, game elimination playback that can be aligned with the human player's perspective, and high humanization and diversity of playing style, so as to truly achieve the purpose of strong humanization of accompanying play, thereby driving the game activity and player participation.

[0302] Since the companion AI will be placed in any area of the farm map, and any area may fight with the player, the model generalization ability is highly required, thus, detailed optimization is needed in the model training battle logic and scene development to realize the training scene covering the whole map. Referring to FIG. 13, the farm map is divided into eight key battle zones, which are 1301, 1302, 1303, 1304, 1305, 1306, 1307 and 1308. When each game session starts, one of the battle zones will be selected according to equal probability to improve the probability of main event occurrence.

[0303] In some embodiments, the two parties (i.e. the first agent and the second agent) in each battle zone are completely symmetrical, and the birth points of the two parties are selected according to a certain logic. In order to avoid that one party is killed by the other party at the beginning of the game, it is required that there is an obstacle between the birth points of each other to ensure sample diversity. At the beginning of the training, a battle zone is randomly selected. For the selected battle zone, a further random selection of the internal birth point combination is performed. When one party dies or the game session is timed out, the single session ends to improve the training efficiency. FIG. 14A is a flowchart of a kind of process for determining the battle configuration logic provided by the embodiments of the present application, which includes the following steps S1401 to S1410, which are explained in detail as follows.

[0304] In step S1401, the selected battle zone is determined.

[0305] In the above example, the battle zone is selected from the above eight key battle zones according to the equal probability principle, and the selected battle zone is obtained. The selected battle zone corresponds to the selected area in other embodiments.

[0306] In step S1402, the center point coordinates of the selected battle zone are determined.

[0307] In some embodiments, the center point coordinates of the selected battle zone can be determined by using geometric center method, barycenter method, point set method, distance weighting method, etc.

[0308] In step S1403, the maximum pathfinding distance of the birth of the first agent is obtained.

[0309] In some embodiments, during the game design and development process, the maximum pathfinding distance of the birth of the first agent is set, i.e. the maximum pathfinding distance of the birth of the first agent is a preset value in advance.

[0310] In step S1404, the first birth area is determined.

[0311] In the embodiments of the present application, the first birth area is determined with the center point coordinates as the center and the maximum pathfinding distance of the birth of the first agent as the radius. Referring to FIG. 14B, 401 is the selected battle zone.

[0312] In step S1405, the navigation grid available points within the first birth area range are sampled to obtain the birth coordinates of the first agent.

[0313] In the embodiments of the present application, the birth coordinates of the first agent can be recorded as the first birth point. Continue to refer to FIG. 14B, 402 is the first birth point.

[0314] In some embodiments, the implementation process of step S1405 is similar to the implementation process of “determining the first available point set in the first birth area” in step S003 and “determining the first initial position of the first agent from the first available point set” in step S004. Therefore, the implementation process of step S1405 can refer to the implementation process of “determining the first available point set in the first birth area” in step S003 and “determining the first initial position of the first agent from the first available point set” in step S004.

[0315] In step S1406, the birth minimum pathfinding distance and the birth maximum pathfinding distance of the second agent are obtained.

[0316] In some embodiments, the birth minimum pathfinding distance and the birth maximum pathfinding distance of the second agent are also preset values.

[0317] In step S1407, the second birth area is determined.

[0318] In the embodiments of the present application, the birth coordinates of the first agent are taken as the center of a circle, the birth minimum pathfinding distance of the second agent is taken as the inner radius of the circle, and the birth maximum pathfinding distance of the second agent is taken as the outer radius of the circle to determine a circular ring, and the circular ring is determined as the second birth area. Continue to refer to FIG. 14B, 403 is the second birth area.

[0319] In step S1408, the navigation grid available points within the second birth area range are sampled to obtain the sampling points.

[0320] In some embodiments, the implementation process of step S1408 is similar to the implementation process of “determining the second available point set in the second birth area” in step S007 and “determining the candidate initial position from the second available point set” in step S008. Therefore, the implementation process of step S1408 can refer to the implementation process of “determining the second available point set in the second birth area” in step S007 and “determining the candidate initial position from the second available point set” in step S008.

[0321] In step S1409, it is determined whether there is an occlusion between the sampling points and the birth coordinates of the first agent.

[0322] In the embodiment of the present application, when there is an occlusion between the sampling point and the birth coordinate of the first agent, the phenomenon that one party is killed at the start of the game match can be avoided, and the sampling point can be used as the birth coordinate of the second agent, and then step S1410 is entered. When there is no occlusion between the sampling point and the birth coordinate of the first agent, the phenomenon that one party is killed at the start of the game match can occur, and the sampling point cannot be used as the birth coordinate of the second agent, and then step S1408 is returned to continue sampling.

[0323] In step S1410, the sampling point is determined as the birth coordinate of the second agent.

[0324] In the embodiment of the present application, the birth coordinate of the second agent can be recorded as the second birth point. Continue to refer to FIG. 14B, 404 is the second birth point.

[0325] In a second aspect, environmental interaction.

[0326] FIG. 15 is a schematic diagram of the overall structure of the model training phase provided in the embodiment of the present application. The game client 1501 refers to a game environment integrated with the AI Service SDK for interface calling. The game client sends a start request to the AI server, and the server side returns corresponding match configuration data according to the training requirements. The game client obtains the data, builds a match, and generates a reinforcement learning agent. Each subsequent request response game frame sample is processed into state data 1502 and sent to the reinforcement learning framework 1503 for interaction. The AI training service predicts the action of the current game request frame, splices the network action into executable action instructions 1504, and returns the executable action instructions 1504 to the client game engine for execution. Then the AI training service continues to wait for the next game frame sample request response. The model training phase will periodically report business side statistical indicators to the training platform to master the training effect. At the end of each game match, the game client will call the relevant logic interface to destroy the current game agent, and automatically start the next round of game match with the same configuration. This is repeated to collect data for the reinforcement learning framework to train.

[0327] In the embodiment of the present application, the model training stage adopts a synchronous inference mode, and a synchronous inference business flow is shown in FIG. 16. There is no human player in each game in the training stage, and there is no various business behavior tree AI and irrelevant game business logic, and only two training battle AIs run in the game kernel (DS), the game business logic is simple and pure, and a large amount of server performance is not needed to consume for collision and ray detection, and the AI is in a synchronous waiting state in the time interval from sending the current frame response of the game to obtaining the action return packet. Among them, the state request 03 is sent by the game kernel 01 to the AI training service 1602, and the action return packet 04 is returned by the AI training service 1602 to the game kernel 01, and the AI training service 1602 corresponds to the first prediction model to be trained in other embodiments. Only after the action return packet is obtained, the game kernel logic will continue to run, so the game kernel in the training stage is a continuous cycle process of the state request 03 and the action return packet 04, and the action request is made every n game frames. The synchronous inference mode of the model training stage can unify the game business logic of the game kernel itself and the reinforcement learning training in the same time line.

[0328] In the embodiment of the present application, the model online stage adopts an asynchronous inference mode, and the asynchronous inference business flow is as shown in FIG. 17. There are a large number of online players in each game in the online stage, and game interaction logic is generated between players and between players and various AIs. If the online stage still adopts a synchronous inference mode, the game kernel will perform a large number of collision and ray detection in the game kernel due to the construction of state data each time the game kernel sends a game frame request response, which will consume most of the server performance. The consumption of server resources will be proportional to the increase in the number of AIs deployed in each game after the online stage, and more seriously, because the game kernel is in a synchronous waiting state during the state request and action return process of the synchronous inference mode each time, the game logic of the business itself will not be operated during this period, and only after the action return is obtained, the game kernel game business logic will continue to run, resulting in a frame freezing phenomenon of the game in the synchronous waiting empty window period (which may be only a few tens of milliseconds). Moreover, this frame freezing phenomenon will be further amplified due to the model inference time and the return delay caused by network fluctuations, which is unacceptable for an FPS game that focuses on high-quality realistic hardcore. Therefore, it is necessary to isolate the game business logic of the game kernel 01 and the logic of the inference service 1702, to ensure that the inference service 1702 does not block the game business logic of the game kernel 01, and to migrate the large amount of collision and ray detection work originally performed on the game kernel 01 side out of the game kernel environment and directly develop it on the inference service 1702. The inference service 1702 will pre-save a large amount of static collision data, perform ray detection query work on the state request 03 sent by the game kernel 01 each time, and return the action return 04 to the game kernel 01, thereby avoiding the large consumption of the performance of the game kernel 01 due to the dynamic ray detection work on the game kernel 01 side each time. In this way, only the logic of the inference service 1702 needs to be concerned about to improve the development efficiency. The inference service 1702 corresponds to the trained first prediction model in other embodiments.

[0329] Based on this, the model online stage adopts an asynchronous inference mode. Asynchronous inference has two timelines. The game kernel side sends basic state request data to the inference service at a fixed interval (every 4 frames). The inference service side performs static collision and ray detection query on the basic state data received according to the first-in-first-out principle, processes the network state data for inference calculation, and then returns the action instruction package to the game kernel side in turn according to the first-in-first-out principle. Since it is an asynchronous inference mode, the game kernel does not need to wait for the action return package after sending the state request response every 4 frames, and continues to run its own game business logic. At the same time, it detects whether there is an action return package every frame, and if there is, it immediately executes the latest action. In addition, if the speed of the action return package on the inference service side is affected by factors such as inference calculation time consumption and network fluctuation delay, the return package speed may not be able to keep up with the sending speed, and as the number of return packages increases, the timeliness of the model's predicted actions will become worse due to the delay of the return package, resulting in the loss of significance of the inference service. Therefore, it is necessary to ensure that the return speed of the inference service is greater than or equal to the sending speed, and at least to achieve a dynamic balance between sending and receiving packages. Based on this, the sending speed can be reduced by increasing the request interval frame number n, or the model structure can be optimized to reduce the inference time consumption to speed up the sending speed. If the number of return package accumulations is too large, a rollback to a behavior tree AI is performed to serve as a bottom line. Asynchronous inference can prevent data from blocking the main thread, ensure that neural network inference time consumption and action return package delay caused by network fluctuations do not affect other game business logic, reduce the occupation of the business side server, and ensure the safety of model inference. If inference fails due to network fluctuations and other factors, the business logic can be switched to a normal behavior tree AI to ensure that the game logic runs stably and normally. This hybrid reinforcement learning access logic has the characteristics of easy use, flexibility, safety, etc.

[0330] In a third aspect, the training side.

[0331] For a first-person shooter game that focuses on tactical games, the game map scene is usually open and diverse, and the gameplay is also different. There are a large number of static, dynamic and environment perception related objects in the three-dimensional scene similar to the real world, and the details are very realistic. The three-dimensional terrain is also complex enough, and the game operation panel also includes a large number of joystick actions that can be executed simultaneously. In such a super large map and three-dimensional complex and variable game environment, it is necessary to train a high-strength accompanying AI that can smoothly move across the map and reasonably use various shelters for battle and pulling. The AI's smooth map running and full-map moving ability is highly required. Therefore, the embodiments of the present application adopt a hybrid pathfinding method with rule enhancement, cooperate with a rich and complete reward and punishment system optimization, and perform self-game iteration training in a two-person zero-sum game scene. Finally, it converges to a Nash equilibrium strategy. At this time, the strength and personification of the model can meet the business objectives.

[0332] In the whole large and complex farm map three-dimensional scene, if only the sixteen direction movement mode is used, the AI is easy to be stuck by various strange terrains, resulting in that in a large number of training games, the neural network focuses on learning how to navigate, thereby ignoring learning the most important battle game pulling ability. In addition, even if the network has focused on learning how to navigate, due to the fact that the training map is too large in size and the terrain is too complex and changeable, the AI is still difficult to move freely on the whole map smoothly without being stuck. More seriously, the battle may occur in any area on the map, and it is easy to appear that the AI is stuck by strange terrain when entering the battle, and finally it is found that the optimal strategy learned by the AI is to wait passively near the birth point for the target enemy to actively find it, and then passively counterattack to win the target enemy approaching by relying on the advantage of pre-positioning and squatting. This strategy obviously does not meet the business goal. It also prolongs the training period and increases the time cost.

[0333] In order to improve the training efficiency while considering the whole map navigation movement effect and battle game ability, the embodiment of the present application adopts a hybrid navigation scheme, uses a centralized training based on rule enhancement and a hybrid execution navigation movement scheme, directly accesses the UE4 bottom layer navigation grid navigation rule as a global movement, and retains the original atomic movement as a local movement. In order to flexibly select when to use global movement and when to use local movement, the neural network adds a hybrid navigation action head. The navigation action output head is used to determine which navigation mode to use. The advantage of the hybrid navigation scheme is that:

[0334] First, rely on the powerful adaptive learning ability of the neural network to freely and flexibly select rule movement or atomic movement. In a non-battle scene, the network learns more about rule movement, and the navigation learning process is weakened. In a battle scene, the network learns more about using atomic movement for local positioning and micro-operation, so that the network focuses on learning the battle game process. In different scenes, the AI has adaptive movement strategies, which is very flexible and controllable, and truly achieves smooth navigation movement in the whole map range, while helping to improve the model strength.

[0335] Second, after using the hybrid navigation movement scheme, the AI is not easy to be stuck by strange terrain, and more battle samples than navigation samples can be generated in the training game, which is conducive to the network focusing more on learning the battle method of mutual game pulling, speeding up the training efficiency and saving the time cost of training.

[0336] The hybrid navigation scheme training process provided by the embodiment of the present application includes: first, a three-finger modeling scheme is adopted, and a three-finger operation rule representing the upper limit of human operation is used to prune (or filter) a large number of illegal and invalid joint actions, so as to obtain double improvement of training efficiency and personification.

[0337] In some embodiments, the network action output head categories and meanings are shown in Table 1, where the Path type action head indicates the selection of rule movement mode or atomic movement mode. The Path type action head further includes the dimension action meaning of "keep the previous movement mode", because the rule movement logic is: first determine the pathfinding target point, and then query the optimal path for the UE 4 to move to the target point based on the pathfinding target point; and then move to the pathfinding target point along the optimal path queried by means of the UE 4 underlying navigation grid, where the pathfinding target point corresponds to the target position in other embodiments, and the optimal path corresponds to the movement path in other embodiments.

[0338] Table 1 Network action output head categories and meanings

[0339] In some embodiments, if the query optimal pathfinding movement path operation consumes server performance, and if the prediction frequency of the network action head is high, the rule movement will be continuously predicted, and the optimal path will be continuously and densely queried in a short time, thereby causing a large amount of server performance overhead in a short time, causing game business logic to lag, and affecting the game experience of online players. In addition, from the perspective of the god, the model may not need to repeatedly predict the rule movement by means of the navigation grid in a short time, because the rule movement by means of the navigation grid mainly occurs in pathfinding in a non-combat scene, at which time the position of the target enemy may not change much, meaning that the optimal path to the target enemy may not change much, and the optimal path queried last time still has timeliness, and the model can still continue to use the optimal path queried last time. And even if the position of the target enemy changes, in a non-combat scene, the model only needs to be able to smoothly move in the whole graph, and is not required to be stuck by strange terrain, so the model can still use the optimal path queried last time.

[0340] Through the hybrid pathfinding scheme provided in the embodiments of the present application, in a non-combat scene, the model does not need to frequently change the optimal path, and can still use the optimal path returned last time. Single query can also avoid unnecessary performance consumption caused by repeated query of the optimal path, so the neural network does not need to frequently predict rule movement instructions in a short time, and only needs to predict rule movement once. In the following time, the rule movement is kept as predicted last time. The optimal path queried by the UE 4 can continue to move, and the rule movement is moved based on the navigation grid, that is, the meaning represented by the Path type action index 2 in Table 1 is to move along the optimal path queried last time by means of the navigation grid.

[0341] In some embodiments, when a certain logic is triggered, such as the agent has reached the position of the pathfinding target point; or encounters the target enemy in advance during the movement along the recently queried optimal path, and enters the battle scene; or has walked along the recently queried optimal path to a time threshold, in the above three cases, the regular movement mode of the navigation grid is interrupted, corresponding to the Path type action index 3 in Table 1, indicating stopping the auxiliary movement. When the next network prediction time comes, the selection of auxiliary regular movement or atomic movement is restarted. In the embodiments of the present application, there is a time threshold constraint every time the navigation grid is used for movement, regardless of whether the pathfinding target point is reached. As long as the time threshold constraint is reached, the regular movement based on the navigation grid is stopped, and the pathfinding mode selection is restarted when the next neural network decision is made. In the whole hybrid pathfinding scheme process, only when the Path type predicted action instruction 1 is performed, the optimal path query work is performed, which consumes the server performance. The subsequent only needs to continuously predict the action instruction 2 to maintain the auxiliary regular movement based on the navigation grid, so as to occupy only a small performance overhead to complete the movement navigation in the whole map range, and also does not affect the various walking game of the agent in the battle scene. The decision exploration advantage of reinforcement learning can be truly maximized to adaptively learn the pathfinding strategy and game strategy in different scenes. In this way, the focus can be placed on how to game and pull, and the pathfinding process in the non-battle scene is weakened, the training efficiency is accelerated, and the model upper limit effect is also improved.

[0342] In the embodiments of the present application, the network structure of the hybrid pathfinding action head is shown in FIG. 18A. The input information 1801 is input to the neural network 1802, and the neural network 1802 is used to predict the pathfinding index 1803. When the pathfinding index is 0, atomic movement is performed, and the action index is (0-15). When the pathfinding index is 1, regular movement based is performed, and the action index is 16. When the pathfinding index is 2, the regular movement based is maintained, and the action index is 16. When the pathfinding index is 3, the regular movement based is stopped, and the action index is (0-15). The input information 1801 corresponds to the state data in other embodiments. Then, when the pathfinding index is 1, the pathfinding target point 1804 and the best path 1805 are determined. When the pathfinding index is 2, the movement along the best path 1805 is continued until the pathfinding target point 1804 is reached. In some embodiments, if the input information 1801 represents a non-battle scene, the pathfinding target point 1804 is reached. If the input information 1801 represents triggering any one of the above three specific logics, the regular movement based is stopped.

[0343] The hybrid pathfinding scheme steps of the embodiments of the present application are as follows:

[0344] Step one, the game match is divided into several time slices according to time, and the starting time of each time slice T is determined by the network to determine the pathfinding mode.

[0345] In the embodiment of the application, the time slice T corresponds to the prediction period in other embodiments. Referring to FIG. 18B, in time slice 1 and time slice 4, the pathfinding mode at the starting time is the atomic movement mode.

[0346] Step two, when the starting point of each time slice T selects the regular movement, the UE 4 queries the optimal path based on the pathfinding target point, and then moves along the optimal path with the help of the navigation grid. Subsequently, the optimal path is maintained within the time slice, and the navigation grid is used to move.

[0347] Referring to time slice 2, time slice 3 and time slice 4 in FIG. 18B, in this way, only one pathfinding path optimization needs to be performed, and the single optimal path query is beneficial to reduce the server performance consumption.

[0348] Step three, within a single time slice, at least one of the following conditions occurs: reaching the pathfinding target point, reaching the time threshold constraint, and one party being seen, the regular movement is immediately ended, and the atomic movement mode is switched to the atomic movement mode in the remaining time of the time slice.

[0349] Referring to FIG. 18B, time slice 2 switches the movement mode from the regular movement mode to the atomic movement mode when the time threshold is reached; time slice 3 switches the movement mode from the regular movement mode to the atomic movement mode when one party is seen; and time slice 4 switches the movement mode from the regular movement mode to the atomic movement mode when the pathfinding target point is reached. Among them, one party being seen corresponds to the first intelligent body being switched from the occlusion scene to the non-occlusion scene in other embodiments.

[0350] Step four, the selection of the pathfinding target point can be output by the model or by the rule. For example, the pathfinding target point can be set as the last target enemy sound perception point.

[0351] FIG. 18A shows the decision logic of the path type network action head, and the input information 1801 includes a multi-modal environment information group, and the meaning of the path type action instruction is shown in Table 1. Among them, the path type network action head corresponds to the hybrid pathfinding action head in other embodiments. Referring to FIG. 18B, the game match is divided into multiple time slices, and the starting time of each time slice is determined by the neural network Path type action head according to the current game situation (i.e. the scene in which the AI is located) to select a specific pathfinding mode. “0” indicates that the entire time slice T is atomic movement; “1+2” indicates that the action instruction “1” is performed first, and then the action instruction “2” is performed, which represents that the optimal path is queried first, and then the navigation grid is used to move along the optimal path; and 3 indicates that the auxiliary regular movement is stopped.

[0352] In some embodiments, FIG. 19 is a flow diagram of hybrid pathfinding according to an embodiment of the present application, which is explained in detail as follows.

[0353] In step S1901, the next time slice is entered.

[0354] In the embodiment of the present application, it means that the current time slice ends and a new time slice is entered.

[0355] In step S1902, it is determined whether the combat scene is entered.

[0356] In the embodiment of the present application, if the non-combat scene is entered, step S1903 is entered; if the combat scene is entered, step S1905A is entered.

[0357] In step S1903, it is determined whether any one of the above three specific logics is met.

[0358] In the embodiment of the present application, if any one of the above three specific logics is not met, it means that the pathfinding target point is not reached at this time, the time threshold constraint is not reached at this time, and one party is not seen at this time, and then step S1904 is entered; if any one of the above three specific logics is met, it means that the pathfinding target point is reached at this time, or the time threshold is reached at this time, the time threshold constraint is met at this time, or one party is seen at this time, and then step S1905D is entered.

[0359] In step S1904, the pathfinding index is obtained.

[0360] In the embodiment of the present application, when the pathfinding index is 0, step S1905A is entered; when the pathfinding index is 1, step S1905B is entered; when the pathfinding index is 2, step S1905C is entered; and when the pathfinding index is 3, step S1905D is entered.

[0361] In step S1905A, the agent is controlled to perform atomic movement, and the flow ends.

[0362] In step S1905B, the agent is controlled to move based on rules.

[0363] In step S1905C, the agent is controlled to keep moving based on rules, and step S1907 is entered.

[0364] In step S1905D, the agent is controlled to stop moving based on rules, and the flow ends.

[0365] In the embodiment of the present application, the above step S1905A, step S1905B, step S1905C and step S1905D are executed according to the pathfinding index.

[0366] In step S1906, the optimal path is determined.

[0367] In the embodiment of the present application, the optimal path is determined based on the current position and the pathfinding target point.

[0368] In step S1907, the current optimal path is maintained.

[0369] In the embodiment of the present application, the optimal path is not updated, and the current optimal path is maintained.

[0370] In step S1908, the pathfinding target point is moved to.

[0371] In the embodiment of the present application, the pathfinding target point is moved to along the optimal path with the aid of the navigation grid.

[0372] In some embodiments, for non-battle scenarios, the neural network outputs a pathfinding target point, and the agent is trained to approach the target point by reinforcement learning. Only when the agent reaches the pathfinding target point, a new pathfinding target point is set. When the agent does not reach the pathfinding target point, the agent is encouraged to approach the pathfinding target point by rewards. This way can also achieve the purpose of pathfinding in non-battle scenarios, and can also avoid the performance consumption of the server caused by multiple path queries.

[0373] Through the above steps S1901 to S1908, atomic movement can be performed in battle scenarios, rule-based movement can be performed in non-battle scenarios, and the movement mode can be matched to the scene to improve the model training efficiency. In addition, when performing rule-based movement, the optimal path only needs to be determined once, which can save computing resources and thus speed up the model training speed.

[0374] FIG. 20 is a structural schematic diagram of a model structure according to an embodiment of the present application. As shown in FIG. 20, the model structure includes a perception part 2001 and a decision part 2002. The perception part 2001 includes environment information 0011, scalar information 0012, depth map information 0013, ring ray information 0014, a first encoder 0015, a second encoder 0016, a third encoder 0017, a fourth encoder 0018, a feature extraction unit 0019, and global information 0020. The decision part 2002 includes an action mask 0021, a policy network 0022, and a value network 0023.

[0375] In the embodiment of the present application, referring to FIG. 20, the first encoder 0015 is configured to encode the environment information 0011 to obtain encoded environment information; the second encoder 0016 is configured to encode the scalar information 0012 to obtain encoded scalar information; the third encoder 0017 is configured to encode the depth map information 0013 to obtain encoded depth map information, wherein the third encoder 0017 can be a Conv2D; and the fourth encoder 0018 is configured to encode the annular ray information 0014 to obtain encoded annular ray information, wherein the fourth encoder 0018 can be a Conv1D.

[0376] Continuing to refer to FIG. 20, the encoded environment information, the encoded scalar information, the encoded depth map information and the encoded annular ray information are spliced to obtain comprehensive encoded information 0024. Then, the feature extraction unit 0019 is used to perform feature extraction on the comprehensive encoded information 0024 to obtain state features, wherein the feature extraction unit 0019 can be a long short-term memory neural network. Then, the state features are input into the policy network 0022, and the policy network 0022 combines the action mask 0021 to predict a set of action instructions 0025.

[0377] Continuing to refer to FIG. 20, the game start triggers the training, the game kernel obtains runtime state data, and the state data is divided into different modal perception data according to Table 2. Different modal perception data are independently extracted by different types of neural networks, which can more accurately and finely represent the current game situation state. The different modal perception data are the environment information, the scalar information, the depth map information and the annular ray information.

[0378] Table 2: State data division result table based on multi-modal environment perception

[0379] Continuing to refer to FIG. 20, the value network 0023 inputs hidden state encoded information output by the feature extraction unit 0019 and global information. The global information is obtained by using all real information of the self and the target enemy. The global input feature including the “cheating” information can reduce the value estimation bias and variance of the value network 0023, which is conducive to the convergence and stability of the policy. This method is only used in the model training stage, because actually only the policy network is needed in the actual online deployment stage. In addition, the autoregressive embedding is introduced into the model design, and continuing to refer to FIG. 20, each layer of action head sampling action is processed by one-hot encoding into an embedding vector, and then the embedding vector is stacked with the input vector of the next layer to serve as the input of the next layer of action head, which forms a kind of causal dependence between the front and the back, so that the decision made by the subsequent action head is more reasonable and natural. The advantages of the model training structure design shown in FIG. 20 are as follows:

[0380] First, the highly complex neural network fuses multi-modal input information such as lists, images, scalars, and Booleans;

[0381] Second, the network action head design through self-recurrent embedding decouples the dependency between the structured action spaces of the FPS game;

[0382] Third, the action mask accelerates training efficiency and also improves the strategy anthropomorphism;

[0383] Fourth, different types of input are activated separately to improve the accuracy of the network's perception of the input.

[0384] Because the original reward and punishment setting makes the agent behave only to target the player, the playing method is mindless and aggressive, and more relies on pre-positioning shooting and ultra-high hit rate to win, lacking the process of spinning and playing, and the agent also does not use cover to drag the battle, the overall anthropomorphism of the battle is low, and it cannot meet the online standard of the business side. Therefore, in the model training process, the embodiment of the present application will join more auxiliary anthropomorphism rewards and punishments to guide the anthropomorphic behavior of the agent based on the original main reward and punishment system.

[0385] In some embodiments, according to the business side playing method, the initial proportion of each reward and punishment coefficient is formulated, which is divided into a main reward and punishment and an auxiliary anthropomorphism reward and punishment. In the initial training period, the agent is encouraged to train for the final victory. In the initial period, the coefficient weight of the main reward and punishment can be adjusted to be high, and the proportion of the anthropomorphism reward and punishment can be adjusted to be low. With the continuous training and the continuous improvement of the model strength, the coefficient proportion of the auxiliary reward and punishment needs to be dynamically adjusted according to the specific performance of the model and the change trend of the corresponding anthropomorphism index. If the agent does not use cover to drag the battle, it is analyzed that the weight of the main kill reward is too large, and the agent only cares about the "greedy" kill reward with a larger value and gives up learning the anthropomorphic behavior guided by the auxiliary reward. Therefore, when the strength of the agent reaches the requirement, the weight of the auxiliary reward needs to be dynamically adjusted to ensure that the agent can further guide the anthropomorphic behavior on the basis of a certain strength.

[0386] In some embodiments, the model performance can also be further optimized through the feedback of the business side experience. The model experience can be regularly carried out during the training process, and the corresponding reward coefficient is adjusted based on the strength or playing method problem obtained through the experience, so as to guide the anthropomorphic multi-style performance. The FPS category game needs to reasonably balance between strength and anthropomorphism. Sometimes, part of the strength can be sacrificed for anthropomorphism. Finally, through continuous experience iterative training, a model that meets the business target and can guarantee the strength and has different anthropomorphic style performance is obtained.

[0387] In some embodiments, the reward and punishment system setting is as shown in Table 3:

[0388] Table 3 reward and punishment system setting table

[0389] The embodiment of the application adopts a self-game iteration process. Since the FPS game is similar to a three-dimensional scene in the real world, the map range is wide enough, and the scene terrain is complex and changeable enough. Therefore, a two-person zero-sum game framework needs to be constructed, and the self-game technology needs to be iterated continuously, and finally converges to the Nash equilibrium strategy. The overall training process includes the following steps one to step eight:

[0390] Step one, the game client built-in behavior tree is used as a battle opponent to train the 0th round model.

[0391] In some embodiments, considering that the users on the product line are relatively active and have large-scale landing battle data, human data can be used to guide the early behavior pattern. The model training stage in the early stage will have a lot of repeated and invalid exploration, and it is even easy to fall into some unpredictable special state, which will cause irreversible long-term impact on strategy learning. The strategy falls into a local optimal point and is difficult to jump out. If the agent can use supervised learning to stand at the starting point of the ability of a human player at the initial moment, and then use reinforcement learning to further explore the strategy, the exploration efficiency can be greatly improved, and the initial strategy learning can be effectively avoided from falling into a local optimum. The subsequent human-like training can also save time and effort.

[0392] Step two, the 0th round model is added to the “evaluation model pool” for evaluation, and the evaluation result is obtained.

[0393] Step three, based on the evaluation result, it is determined whether the 0th round model is stored in the “opponent model pool”.

[0394] Step four, a past model is sampled from the “opponent model pool” according to a certain rule as an opponent to train the Kth round model until the best response is reached.

[0395] In the embodiment of the application, the uniform sampling rule, the elite sampling rule, etc. can be used.

[0396] Step five, the Kth round model is added to the “evaluation model pool” for evaluation, and the Kth evaluation result is obtained.

[0397] Step six, based on the Kth evaluation result, it is determined whether the Kth round model is stored in the “opponent model pool”.

[0398] Step seven, return to step four to train the K+1th round model.

[0399] Step eight, until the training converges to the Nash equilibrium strategy, stop the training iteration.

[0400] The embodiment of the application adopts self-game reinforcement learning training based on a hybrid pathfinding scheme, gives play to the decision-making exploration advantage of reinforcement learning, centrally trains a neural network, and performs hybrid movement at the bottom layer. According to the self-adaptive and flexible selection of a movement mode according to a battle situation, not only can the full-map pathfinding movement and the battle game ability be taken into account, but also the training efficiency is greatly improved. In addition, in combination with a complete and detailed auxiliary personification reward and punishment system, a high-level play-along AI that meets the business requirements can be produced in a super-large map and a complex and diverse three-dimensional terrain scene, and the business team can be helped to more efficiently generate content.

[0401] Based on multi-round self-game iterative training, the reinforcement learning play-along AI trained by the embodiment of the application has reached the business target in terms of strength and personification. The advantages of the reinforcement learning play-along AI include:

[0402] First, the recognition of complex obstacles is more flexible. Compared with the behavior tree version, the reinforcement learning AI can recognize the position of a shelter and go to the shelter to avoid without pre-constructing a shelter point, and can use various unconventional shelters to avoid battles.

[0403] Second, the behavior is flexible and variable, and different battle performances can be achieved in the same wrestling scene, which can improve the repeat playability.

[0404] Third, the behavior space completely covers the player's behavior space, and all actions that can be performed by the player are completed, so that places that cannot be reached by the behavior tree and operations that cannot be made by the behavior tree are completed.

[0405] Fourth, the firing personification and the behavior personification have reached a level that the behavior tree cannot reach, and the game elimination playback can also be aligned with the human player's perspective, and the purpose of the play-along AI is truly achieved.

[0406] FIG. 21A is the distribution of the activity area of the agent trained based on the hybrid movement pathfinding scheme in the map, and FIG. 21B is the distribution of the activity area of the AI trained based on the atomic movement pathfinding scheme in the map. As shown in FIGS. 21A and 21B, when the hybrid movement pathfinding scheme is adopted, the activity area of the agent (i.e., the model) is more dispersed, and the full-map coverage range is also wider, which highlights the improvement of the hybrid movement pathfinding scheme on the full-map movement navigation ability of the AI.

[0407] Referring to FIG. 22, curve 1 is the behavior tree win rate of the AI trained based on the hybrid movement pathfinding scheme, and curve 2 is the behavior tree win rate of the AI trained based on the atomic movement pathfinding scheme. As shown in curve 1 and curve 2, when the agent is trained based on the hybrid pathfinding scheme, a higher win rate can be quickly reached in the same time, which indicates that the AI model trained by the hybrid scheme has a higher upper limit.

[0408] Referring to FIG. 23, curve 3 is the draw rate of the AI combat behavior tree trained based on the hybrid mobile pathfinding scheme, and curve 4 is the draw rate of the AI combat behavior tree trained based on the atomic mobile pathfinding scheme. As can be seen from curve 3 and curve 4, the AI trained based on the atomic mobile scheme has a higher draw rate, which indicates that the strategy converges to a more and more passive style, which is manifested in that the agent is unwilling to take the initiative to attack and only passively waits for the enemy to come to fight, which does not meet the expectations of the project team for the agent. The AI trained based on the hybrid mobile scheme has a lower draw rate, but the agent has a higher win rate. In combination with FIG. 21A, it is further indicated that the agent trained based on the hybrid mobile scheme has enhanced initiative, a wider active area, and improved pathfinding ability, navigation ability, and model strength.

[0409] It can be understood that in the embodiments of the present application, the related data such as state data, a first prediction model to be trained, a main reward coefficient, and an auxiliary reward coefficient are involved, and the collection, use, and processing of the related data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0410] The following continues to illustrate an exemplary structure of the implementation of the model training apparatus 455 provided in the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 2, the software module stored in the model training apparatus 455 in the memory 450 can include:

[0411] The first acquisition module 4551 is configured to acquire state data of a first agent in a virtual scene and a first prediction model to be trained corresponding to the first agent. The first prediction model to be trained is used to control the first agent to move based on the state data of the first agent. The instruction prediction module 4552 is configured to predict the state data by using the first prediction model to be trained to obtain a set of action instructions, the set of action instructions including a pathfinding manner and a plurality of action information. The execution module 4553 is configured to control the first agent to move based on the pathfinding manner and perform a corresponding action based on each of the action information to obtain an execution result. The adjustment module 4554 is configured to adjust parameters of the first prediction model to be trained based on reward information of a plurality of execution results to obtain a trained first prediction model, wherein the reward information is determined based on the plurality of execution results, a main reward coefficient, and an auxiliary reward coefficient.

[0412] In some embodiments, the first prediction model comprises a feature extraction module, a pathfinding prediction module, and a plurality of action prediction modules, and the instruction prediction module 4552 is further configured to: perform feature extraction on the state data by using the feature extraction module to obtain state features; for each action prediction module, perform prediction based on the state features by using the action prediction module to obtain action information of the first agent; perform prediction based on the state features by using the pathfinding prediction module to obtain a movement manner of the first agent; and construct the action instruction set based on the movement manner and the action information.

[0413] In some embodiments, the state data comprises environment information, scalar information, depth map information, and annular ray information, the feature extraction module comprises an encoding unit and a feature extraction unit, and the instruction prediction module 4552 is further configured to: perform encoding processing on the environment information, the scalar information, the depth map information, and the annular ray information respectively by using the encoding unit to obtain encoded environment information, encoded scalar information, encoded depth map information, and encoded annular ray information; splice the encoded environment information, the encoded scalar information, the encoded depth map information, and the encoded annular ray information to obtain comprehensive encoded information; and perform feature extraction on the comprehensive encoded information by using the feature extraction unit to obtain the state features.

[0414] In some embodiments, the instruction prediction module 4552 is further configured to: obtain a preset action sequence table, the action sequence table being used to represent the arrangement order between actions; determine a first action identifier located in front of a pathfinding action identifier based on the action sequence table; obtain first action information corresponding to the first action identifier, the first action information being predicted by using an action prediction module corresponding to the first action identifier; encode the first action information to obtain first encoded action information; and perform prediction based on the state features and the first encoded action information by using the pathfinding prediction module to obtain a movement manner of the first agent.

[0415] In some embodiments, the instruction prediction module 4552 is further configured to: perform prediction on the state features and the first encoded action information by using the pathfinding prediction module according to a prediction period to obtain a target pathfinding index; determine a target movement manner corresponding to the target pathfinding index based on a preset index mapping table; and determine the target movement manner as the movement manner of the first agent.

[0416] In some embodiments, the target routing index is one of a first routing index and a second routing index, and when the target routing index is the second routing index, the moving manner of the first agent is determined as a rule-based moving manner; the execution module 4553 is further configured to: obtain a target position and a current position of the first agent; determine a moving path of the first agent based on the target position and the current position; control the first agent to move in the rule-based moving manner based on the moving path; and control the first agent to perform each action based on each action information.

[0417] In some embodiments, the execution module 4553 is further configured to: during the process of controlling the first agent to move in the rule-based moving manner, when it is determined that a moving manner switching condition is met, update the target routing index to the first routing index; and switch the moving manner of the first agent to an atomic moving manner.

[0418] Control the first agent to move atomically.

[0419] In some embodiments, the execution module 4553 is further configured to: when it is determined that the first agent moves to the target position, determine that the moving manner switching condition is met; or when it is determined that the first agent switches from an occlusion scene to a non-occlusion scene, determine that the moving manner switching condition is met, wherein the occlusion scene represents that there is an occlusion between the first agent and the second agent, and the non-occlusion scene represents that there is no occlusion between the first agent and the second agent; or obtain a moving duration of the first agent moving in the rule-based moving manner, and when the moving duration reaches a duration threshold, determine that the moving manner switching condition is met, wherein the duration threshold is less than the prediction period.

[0420] In some embodiments, the instruction prediction module 4552 is further configured to: obtain a second action identifier corresponding to the action prediction module; determine a third action identifier located in front of the second action identifier based on the action sequence table; obtain second action information corresponding to the third action identifier, the second action information being predicted by an action prediction module corresponding to the third action identifier;

[0421] Encode the second action information to obtain second encoded action information; and predict, by the action prediction module corresponding to the second action identifier, the state feature and the second encoded action information to obtain the action information of the first agent.

[0422] In some embodiments, the instruction prediction module 4552 is further configured to: obtain a preset action mask, the action mask being matched with an interaction rule in the virtual scene, the action mask being used to limit an action selection space of the first agent; construct the movement manner and the respective action information into a candidate instruction set; and perform mask processing on the candidate instruction set by using the action mask to obtain the action instruction set.

[0423] In some embodiments, the action information includes a first flag value used to represent whether to perform an action, and the instruction prediction module 4552 is further configured to: for each of the action information in the candidate instruction set, obtain a mask value corresponding to the action information from the action mask; when the mask value corresponding to the action information is a first preset value and the first flag value in the action information is a second preset value, update the first flag value in the action information to the first preset value, wherein when the first flag value is the first preset value, it represents not performing the action corresponding to the action information.

[0424] In some embodiments, the adjustment module 4554 is further configured to: determine a reward function based on the main reward coefficient and the auxiliary reward coefficient, wherein the auxiliary reward coefficient is used to represent a reward coefficient assigned when the first agent performs a preset humanization action; determine reward information of each execution result by using the reward function; accumulate the reward information of each execution result to obtain accumulated reward, and back-propagate the accumulated reward to the first prediction model to be trained, and adjust parameters of the first prediction model to be trained based on the accumulated reward until a training end condition is reached to obtain the trained first prediction model.

[0425] In some embodiments, the model training apparatus 455 further includes: a first determination module configured to determine a selected region from respective sub-regions covered by a virtual scene corresponding to a game match based on an equiprobability principle in response to a game match start instruction; a second determination module configured to determine a first center point of the selected region; and a third determination module configured to determine a first birth region based on the first center point and a preset first maximum path searching distance, and determine a first available point set in the first birth region.

[0426] A first generation module is configured to determine a first initial position of the first agent from the first available point set, and generate the first agent at the first initial position.

[0427] In some embodiments, the model training apparatus 455 further includes a fourth determination module configured to determine the first initial position as a second center point; a second acquisition module configured to acquire a preset second maximum path searching distance corresponding to the second agent and a preset second minimum path searching distance corresponding to the second agent; a fifth determination module configured to determine a second birth area based on the second center point, the preset second maximum path searching distance and the preset second minimum path searching distance, and determine a second available point set in the second birth area; a sixth determination module configured to determine a candidate initial position from the second available point set; and a second generation module configured to determine the candidate initial position as a second initial position of the second agent when there is an obstacle between the candidate initial position and the first initial position, and generate the second agent at the second initial position.

[0428] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the model training method provided in the embodiment of the present application.

[0429] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored in the computer readable storage medium. When the executable instructions are executed by a processor, the processor executes the model training method provided in the embodiment of the present application, for example, the model training method shown in FIG. 3 and FIG. 12.

[0430] In some embodiments, the computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM, and the like. The computer readable storage medium can also be various devices including one or any combination of the above storage devices.

[0431] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0432] By way of example, executable instructions can correspond to a file in a file system, but in many cases can not. An executable instruction can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions.

[0433] By way of example, executable instructions can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0434] The above description is provided as an example only and is not for the purpose of limiting the scope of the application. Any modification, equivalent replacement and improvement made within the spirit and the scope of the application shall fall into the scope of the protection of the application.

Claims

1. A model training method applied to a computer device, the method comprising: obtaining state data of a first agent in a virtual scene and a first prediction model to be trained corresponding to the first agent; the first prediction model to be trained is configured to control the first agent to move based on the state data of the first agent; using the first prediction model to be trained to predict the state data to obtain a set of action instructions, the set of action instructions comprising a pathfinding mode and a plurality of action information; controlling the first agent to move based on the pathfinding mode and perform corresponding actions based on each of the action information to obtain an execution result; adjusting parameters of the first prediction model to be trained based on reward information of a plurality of execution results to obtain a trained first prediction model, wherein the reward information is determined based on the plurality of execution results, a main reward coefficient and an auxiliary reward coefficient.

2. The method of claim 1, wherein, the first prediction model comprises a feature extraction module, a pathfinding prediction module and a plurality of action prediction modules, and the using the first prediction model to be trained to predict the state data to obtain a set of action instructions comprising a pathfinding mode comprises: using the feature extraction module to extract features from the state data to obtain state features; for each action prediction module, using the action prediction module to predict based on the state features to obtain action information of the first agent; using the pathfinding prediction module to predict based on the state features to obtain a pathfinding mode of the first agent; based on the pathfinding mode and each action information, constructing the set of action instructions.

3. The method of claim 1 or 2, wherein, the state data comprises environmental information, scalar information, depth map information and annular ray information, the feature extraction module comprises an encoding unit and a feature extraction unit, and the using the feature extraction module to extract features from the state data to obtain state features comprises: using the encoding unit to respectively encode the environmental information, the scalar information, the depth map information and the annular ray information to obtain encoded environmental information, encoded scalar information, encoded depth map information and encoded annular ray information; splicing the encoded environmental information, the encoded scalar information, the encoded depth map information and the encoded annular ray information to obtain comprehensive encoded information; using the feature extraction unit to extract features from the comprehensive encoded information to obtain the state features.

4. The method according to any one of claims 1 to 3, wherein, the using the pathfinding prediction module to predict based on the state features to obtain a pathfinding mode of the first agent comprises: obtaining a preset action sequence table, the action sequence table being configured to represent an arrangement order between actions; based on the action sequence table, determining a first action identifier located in front of a pathfinding action identifier sequence; obtaining first action information corresponding to the first action identifier, the first action information being predicted by using an action prediction module corresponding to the first action identifier; encoding the first action information to obtain first encoded action information; The pathfinding prediction module is used to predict based on the state feature and the first encoded action information to obtain a pathfinding manner of the first agent.

5. The method according to any one of claims 1 to 4, wherein, The pathfinding prediction module is used to predict based on the state feature and the first encoded action information to obtain a pathfinding manner of the first agent. According to a prediction period, the pathfinding prediction module is used to predict the state feature and the first encoded action information to obtain a target pathfinding index; Based on a preset index mapping table, a target pathfinding manner corresponding to the target pathfinding index is determined; The target pathfinding manner is determined as the pathfinding manner of the first agent.

6. The method according to any one of claims 1 to 5, wherein, The target pathfinding index is one of a first pathfinding index and a second pathfinding index, and when the target pathfinding index is the second pathfinding index, the pathfinding manner of the first agent is determined as a rule-based pathfinding manner; the control of the first agent to move based on the pathfinding manner and to perform corresponding actions based on each action information includes: A target position and a current position of the first agent are obtained; Based on the target position and the current position, a movement path of the first agent is determined; Based on the movement path, the first agent is controlled to move in a rule-based pathfinding manner, and the first agent is controlled to perform each action based on each action information.

7. The method according to any one of claims 1 to 6, wherein, The method further includes: In the process of controlling the first agent to move in a rule-based pathfinding manner, when it is determined that a pathfinding manner switching condition is met, the target pathfinding index is updated to the first pathfinding index; The pathfinding manner of the first agent is switched to an atomic movement pathfinding manner; The first agent is controlled to move atomically.

8. The method according to any one of claims 1 to 7, wherein, The method further includes: When it is determined that the first agent moves to the target position, it is determined that the pathfinding manner switching condition is met; or When it is determined that the first agent switches from an occlusion scene to a non-occlusion scene, it is determined that the pathfinding manner switching condition is met, wherein the occlusion scene represents that there is an occlusion between the first agent and the second agent, and the non-occlusion scene represents that there is no occlusion between the first agent and the second agent; or A movement duration of the first agent moving in a rule-based pathfinding manner is obtained, and when the movement duration reaches a duration threshold, it is determined that the pathfinding manner switching condition is met, wherein the duration threshold is less than the prediction period.

9. The method according to any one of claims 1 to 8, wherein, The action prediction module is used to predict based on the state feature to obtain the action information of the first agent, including: A second action identifier corresponding to the action prediction module is obtained; Based on the action sequence table, a third action identifier located in front of the second action identifier is determined; Second action information corresponding to the third action identifier is obtained, and the second action information is predicted by using an action prediction module corresponding to the third action identifier; The second action information is encoded to obtain second encoded action information; The corresponding action prediction module predicts based on the state feature and the second encoded action information to obtain action information of the first agent.

10. The method according to any one of claims 1 to 9, wherein, The constructing of the action instruction set based on the pathfinding mode and each action information comprises: acquiring a preset action mask, the action mask matching an interaction rule in the virtual scene, and the action mask being used to limit an action selection space of the first agent; constructing the pathfinding mode and the each action information into a candidate instruction set; mask processing the candidate instruction set using the action mask to obtain the action instruction set.

11. The method according to any one of claims 1 to 10, wherein, The action information comprises a first flag value used to represent whether to perform an action, and the mask processing of the candidate instruction set using the action mask comprises: for each action information in the candidate instruction set, acquiring a mask value corresponding to the action information from the action mask; when the mask value corresponding to the action information is a first preset value and the first flag value in the action information is a second preset value, updating the first flag value in the action information to the first preset value, wherein when the first flag value is the first preset value, it represents not performing the action corresponding to the action information.

12. The method according to any one of claims 1 to 11, wherein, The adjusting of the parameters of the first prediction model to be trained based on the reward information of the plurality of execution results to obtain a trained first prediction model comprises: determining a reward function based on the main reward coefficient and the auxiliary reward coefficient, wherein the auxiliary reward coefficient is used to represent a reward coefficient assigned when the first agent performs a preset humanization action; determining the reward information of each execution result using the reward function; accumulating the reward information of each execution result to obtain a cumulative reward, back-propagating the cumulative reward to the first prediction model to be trained, and adjusting the parameters of the first prediction model to be trained based on the cumulative reward until a training end condition is reached to obtain the trained first prediction model.

13. The method of any one of claims 1 to 12, wherein, The method further comprises: in response to a game session start instruction, determining a selected region from each sub-region covered by a virtual scene corresponding to the game session based on an equal probability principle; determining a first center point of the selected region; determining a first birth region based on the first center point and a preset first maximum pathfinding distance, and determining a first available point set in the first birth region; determining a first initial position of the first agent from the first available point set, and generating the first agent at the first initial position.

14. The method of any one of claims 1 to 13, wherein, The method further comprises: determining the first initial position as a second center point; acquiring a preset second maximum pathfinding distance corresponding to the second agent and a preset second minimum pathfinding distance corresponding to the second agent; determining a second birth region based on the second center point, the preset second maximum pathfinding distance, and the preset second minimum pathfinding distance, and determining a second available point set in the second birth region; determining a candidate initial position from the second available point set; When there is an occlusion between the candidate initial position and the first initial position, the candidate initial position is determined as a second initial position of the second agent, and the second agent is generated at the second initial position.

15. A model training apparatus, the apparatus comprising: a first obtaining module configured to obtain state data of a first agent in a virtual scene and a first prediction model to be trained corresponding to the first agent; the first prediction model to be trained is used to control the first agent to move based on the state data of the first agent; an instruction prediction module configured to predict the state data by using the first prediction model to be trained to obtain a set of action instructions, the set of action instructions including a pathfinding manner and a plurality of action information; an execution module configured to control the first agent to move based on the pathfinding manner and perform a corresponding action based on each of the action information to obtain an execution result; an adjustment module configured to adjust parameters of the first prediction model to be trained based on reward information of a plurality of execution results to obtain a trained first prediction model, wherein the reward information is determined based on the plurality of execution results, a main reward coefficient and an auxiliary reward coefficient.

16. A computer device, the computer device comprising: a memory configured to store computer executable instructions; a processor configured to execute the computer executable instructions stored in the memory to implement the model training method according to any one of claims 1 to 14.

17. A computer readable storage medium storing computer executable instructions, the computer executable instructions being executed by a processor to implement the model training method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program or computer executable instructions, the computer program or computer executable instructions being executed by a processor to implement the model training method according to any one of claims 1 to 14.

Images