Modeling method and device of intelligent agent, equipment and storage medium

Abstract

Embodiments of the present application disclose a method for modeling an agent, the method comprising: obtaining game data of a target game scene, the game data comprising at least game scene data, and behavior observation data of all agents pushed by a preset host client in the target game scene; determining visual data of each agent according to the game scene data and the behavior observation data; determining target action instructions of each agent according to the visual data and an action decision model; sending the target action instructions to a client corresponding to each agent, and returning to the step of obtaining the game data and the behavior observation data until the action decision model converges, thereby completing modeling of the agent in the target game scene. In the above manner, the visual data of the agent is dynamically rendered in the AI server, thereby reducing the problem of wasting bandwidth in transmitting the visual data through the network, and the action decision model of the reinforcement learning training makes the agent behavior more human-like.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for modeling intelligent agents. Background Technology

[0002] With the development of artificial intelligence (AI) in recent years, deep learning technology is increasingly being applied to daily life, and the demands of various application scenarios are placing greater challenges on deep learning technology. We can verify our deep learning technology by building virtual environments, constructing agents within these environments. Each agent uses a set of deep learning techniques to complete the process from observation to action decision-making, while simultaneously optimizing the agent model within the virtual environment. In current practical applications, video game scenarios are suitable as verification virtual environments, and game characters can serve as carriers of artificial intelligence. Currently, an increasing number of game genres are combining with AI technology to bring better experiences to players, such as card games, MOBAs, and FPS games. While the aforementioned common game genres often have relatively clear rules and gameplay, another major game genre—sandbox games (e.g., Minecraft)—is different. It lacks clear gameplay and objectives, and its creativity and flexibility have made it widely popular among players, especially 3D games.

[0003] Therefore, existing agent modeling methods are not suitable for agent modeling in open-world gameplay such as sandboxes. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, device, and storage medium for modeling intelligent agents, which can solve the problem in the prior art of lacking effective means for modeling intelligent agents in open-world games.

[0005] To achieve the above objectives, the first aspect of the present invention provides a method for modeling intelligent agents, the method being applied to an AI server, wherein the number of intelligent agents is at least one, and each intelligent agent corresponds one-to-one with a client, the method comprising:

[0006] The game data of the target game scene in the AI ​​server is obtained, as well as the behavior observation data of all intelligent agents pushed by the host client in the target game scene. The game data is used to reflect the information of the players and the scene in the target game scene, and the game data includes at least game scene data.

[0007] Based on the game scene data and the behavior observation data, visual data for each agent is determined. The visual data is used to reflect the object information within the visual range of the agent in the target game scene.

[0008] Based on the visual data and the action decision model, a target action instruction is determined for each agent. The target action instruction is used to control the agent to perform actions that conform to the game's survival rules. The action decision model is a model trained using reinforcement learning.

[0009] The target action command is sent to the client corresponding to each agent, and the process of obtaining game data of the target game scene in the AI ​​server and behavioral observation data of all agents pushed by the preset host client in the target game scene is returned until the action decision model converges, thus completing the modeling of agents in the target game scene.

[0010] In one feasible implementation, the behavioral observation data includes the agent's position and orientation data, and the game scene data includes static map data and dynamic construction data of the target game scene. Then, determining the visual data of each agent based on the game scene data and the behavioral observation data includes:

[0011] The visual range of each intelligent agent is determined based on the preset boundary range, the location data, and the orientation data;

[0012] Scene rendering is performed based on the visual range, the static map data, and the dynamic construction data to obtain depth map data within the visual range of each agent, and the depth map data is used as the visual data of the agent.

[0013] In one feasible implementation, the visual data includes at least the type of obstacle and the distance between the obstacle and the agent. Then, based on the visual data and the action decision model, a target action instruction for each agent is determined, including:

[0014] The type of obstacle and the distance between the obstacle and the agent are input into the action decision model to determine the target action instruction for each agent.

[0015] In one feasible implementation, the game data further includes agent reward data and agent attribute data. Then, determining the target action instruction for each agent based on the visual data and the action decision model further includes:

[0016] The visual data, game scene data, character reward data, and character attribute data are input into the action decision model to determine the target action instruction for each agent.

[0017] In one feasible implementation, the step of acquiring game data of the target game scene from the AI ​​server, and behavioral observation data of all intelligent agents pushed by the preset host client in the target game scene, further includes:

[0018] Receive a communication establishment request sent by the client;

[0019] In response to the communication request, a communication connection is established between the client and the AI ​​server;

[0020] After successfully establishing a communication connection between the client and the AI ​​server, if the client is the first client to successfully establish a communication connection with the AI ​​server, then the client is determined to be the target client; if the client is not the first client to successfully establish a communication connection with the AI ​​server, then the client is determined to be the passenger client.

[0021] In one feasible implementation, the method further includes:

[0022] Send a room creation request to the host client and receive the room creation result returned by the host client;

[0023] When the room creation result is successful, the room is initialized and configured according to the room identifier and training requirements in the room creation result to obtain the target game scene. The training requirements include the game scene, agent training scale, and agent capability attributes.

[0024] In one feasible implementation, the initial configuration process for the room, based on the room identifier and training requirements in the room creation result, further includes:

[0025] Send a room join request to the passenger aircraft client, the room join request including the room identifier;

[0026] Receive the room joining result returned by the passenger aircraft client;

[0027] Count the total number of client machines that successfully joined the room in the room joining results;

[0028] When the total number is not less than the training scale of the agent, the game start command sent by the host client is received, and the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all agents pushed by the host client in the target game scene are executed.

[0029] To achieve the above objectives, a second aspect of the present invention provides a modeling apparatus for intelligent agents, the apparatus comprising:

[0030] Data acquisition module: used to acquire game data of the target game scene in the AI ​​server, as well as behavioral observation data of all intelligent agents pushed by the preset host client in the target game scene. The game data is used to reflect the information of players and scene in the target game scene, and the game data includes at least game scene data.

[0031] Visual determination module: used to determine the visual data of each agent based on the game scene data and the behavior observation data, wherein the visual data is used to reflect the object information within the visual range of the agent in the target game scene;

[0032] Action Decision Module: Used to determine the target action instruction for each agent based on the visual data and the action decision model. The target action instruction is used to control the agent to perform actions that conform to the game survival rules. The action decision model is a model trained using reinforcement learning.

[0033] Action execution module: Used to send the target action instruction to the client corresponding to each of the intelligent agents, and return to execute the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all intelligent agents pushed by the preset host client in the target game scene, until the action decision model converges, thus completing the modeling of the intelligent agents in the target game scene.

[0034] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps shown in the first aspect and any feasible implementation.

[0035] To achieve the above objectives, a fourth aspect of the present invention provides a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps shown in the first aspect and any feasible implementation.

[0036] The embodiments of the present invention have the following beneficial effects:

[0037] This invention provides a method for modeling intelligent agents. The method is applied to an AI server, with at least one intelligent agent and a one-to-one correspondence between intelligent agents and clients. The method includes: acquiring game data of a target game scene from the AI ​​server, and behavioral observation data of all intelligent agents pushed by a preset host client in the target game scene. The game data reflects information about players and the scene in the target game scene, and includes at least game scene data; determining visual data for each intelligent agent based on the game scene data and behavioral observation data. The visual data reflects information about objects within the visual range of the intelligent agent in the target game scene; determining target action instructions for each intelligent agent based on the visual data and an action decision model. The target action instructions control the intelligent agent to perform actions that conform to the game's survival rules. The action decision model is a model trained using reinforcement learning; sending the target action instructions to the client corresponding to each intelligent agent, and returning to the steps of acquiring game data of the target game scene from the AI ​​server and behavioral observation data of all intelligent agents pushed by a preset host client in the target game scene, until the action decision model converges, thus completing the modeling of the intelligent agents in the target game scene. The above approach enables the visual data of agents in open worlds to be dynamically rendered and generated on AI servers. This reduces the bandwidth wastage and performance impact caused by transmitting agent visual data over the network due to the complexity of freely constructed scenes in open worlds. It meets the performance requirements of online access while also controlling the cost of offline training. Furthermore, the action decision-making model trained using reinforcement learning can make action decisions for the agent, resulting in more human-like behavior. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] in:

[0040] Figure 1 This is an application environment diagram of a modeling method for an intelligent agent according to an embodiment of the present invention;

[0041] Figure 2 This is a flowchart of a modeling method for an intelligent agent according to an embodiment of the present invention;

[0042] Figure 3 This is another flowchart of a modeling method for an intelligent agent according to an embodiment of the present invention;

[0043] Figure 4This is an interactive flowchart of the training process of a modeling method for an intelligent agent according to an embodiment of the present invention;

[0044] Figure 5 This is a structural block diagram of a modeling device for an intelligent agent according to an embodiment of the present invention;

[0045] Figure 6 This is a structural block diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figure 1 , Figure 1 This is an application environment diagram of a modeling method for an intelligent agent according to an embodiment of the present invention. Figure 1 This is a modeling system for intelligent agents, comprising clients and servers. Specifically, the servers include an AI server (AIServer) 110 and a game cloud server 140, while the clients include a host client and a passenger client. The clients and servers have a communication connection and can interact with each other. The modeling method for intelligent agents is applied to the AI ​​server. Each client corresponds one-to-one with an intelligent agent in the game. An intelligent agent is an important concept in the field of artificial intelligence; any software or hardware entity capable of autonomously acting and interacting with its environment can be abstracted as an intelligent agent. The 3D scene can be a real-world 3D scene (i.e., a 3D real-world scene) or a virtual 3D scene (i.e., a 3D virtual scene). It can be understood that if the 3D scene is a 3D virtual scene, the intelligent agent can be a 3D virtual character model implemented through software; if the 3D scene is a 3D real-world scene, the intelligent agent can be an AI-based hardware entity, such as a physical AI robot. The shape, form, and type of the obstacle 104 are not limited; any object that can obstruct the movement of the intelligent agent in the 3D scene can be considered an obstacle.

[0048] It should be noted that the AI ​​server is used to execute the agent modeling method in this application. When training is required, the AI ​​server is the first to start. Each client, based on the given Internet Protocol (IP) and port configuration, actively connects to the AI ​​server after it starts. The AI ​​server determines which client connected first and sends a room creation request to that client, making that game client the host client, which monitors and provides feedback on the agent's action execution during subsequent training. Other game clients that connect to the AI ​​server act as guest clients; the AI ​​server only needs to send room joining requests to these guest clients.

[0049] The overall architecture of this invention is mainly divided into two parts: AIServer and game service. The game service is further divided into client and cloud service, which is determined by the game's architecture. Open-world sandbox games like Minecraft use a room-based map selection method. In this way, the game client of the room owner acts as the host, and the other players joining the room act as guests. All clients need to interact with the backend game cloud service to unify room information, download maps, and perform other operations. Therefore, the AIServer needs to connect to multiple game clients and exchange data.

[0050] Please see Figure 2 , Figure 2 This is a flowchart of a modeling method for an intelligent agent according to an embodiment of the present invention, such as... Figure 2 The method is applied to an AI server, and the number of intelligent agents is at least one, with each intelligent agent corresponding to a client. The method includes at least the following steps:

[0051] 201. Obtain game data of the target game scene in the AI ​​server, as well as behavioral observation data of all intelligent agents pushed by the host client in the target game scene. The game data is used to reflect the information of the players and the scene in the target game scene, and the game data includes at least game scene data.

[0052] It should be noted that in the modeling method shown in this embodiment, the number of agents is at least one; that is, there can be one or more agents to be trained, and this embodiment does not impose any limitation. Each agent corresponds one-to-one with a client, allowing different clients to control different agents to execute the model's decision-making actions. Therefore, this application is particularly suitable for agent modeling in sandbox games such as Minecraft. Furthermore, because sandbox games have many game scenarios, and the gameplay and game data differ between scenarios, this application can model agents for different game scenarios, enabling agents to meet the needs of different game scenarios.

[0053] Therefore, during modeling, it is necessary to obtain game data of the target game scene selected by the user, as well as behavioral observation data of the intelligent agent by the host client, in order to model the intelligent agent in the target game scene and obtain an intelligent agent model that can survive autonomously in the target game scene.

[0054] The game data reflects information about the players and the game environment in the target game scene. Player information includes, but is not limited to, information about other agents in the target game scene. Scene information includes, but is not limited to, map data of the target game scene. This map data includes static map data and dynamic building data. Taking Minecraft as an example, players place blocks on the game map to build items. Therefore, static map data can be understood as the initial map data of the game, and dynamic building data is the building data of the players in the game. Behavioral observation data consists of the position and orientation data of the agents in the target game scene.

[0055] For example, game data may also include: global information, agent revenue data, and agent attribute data. Global information includes, but is not limited to, match information, team information, and settlement information; wherein, match information includes, but is not limited to, current time, total time, and game mode. Team information includes, but is not limited to, the number of players in each faction and their survival status, etc. In addition, for competitive maps, it may be a two-way battle, and for open maps, it may be divided into teams according to NPC factions. Settlement information includes, but is not limited to, whether the game has ended, the information of the winning side, and the overall score, etc., and the settlement information may have different victory conditions depending on the game mode. For example, in competitive games, the score may be based on the total number of kills by both sides and the final survival; in racing games, it may be based on the first to arrive.

[0056] Agent benefit data reflects the agent's survival benefits in the game; therefore, agent benefit data includes, but is not limited to, survival information, backpack information, equipment information, and inventory information. Survival information includes, but is not limited to, health, hunger, oxygen, fatigue, health regeneration speed, and hunger regeneration speed. Backpack information includes, but is not limited to, backpack capacity, item ID, type, quantity, and whether it is currently in use. Equipment information includes, but is not limited to, equipment ID, type, and durability. Inventory information includes, but is not limited to, item ID, type, whether it is currently in use, and for firearms, reload time and remaining ammo; for some items that can be charged, there is a charge bar.

[0057] Agent attribute data reflects the agent's basic attributes, including but not limited to identification information, location information, ability information, and voice information. Identification information includes, but is not limited to, player ID, whether it is an agent, and its team ID; location information includes, but is not limited to, coordinates, orientation, speed, and movement status; ability information includes, but is not limited to, defense, attack, speed, ranged attack, and magical attack; and voice information includes, but is not limited to, type and coordinates.

[0058] Among these, agent attribute data, agent reward data, and behavioral observation data can be collectively referred to as player information. With the above data, the agent can obtain information that aligns with human players and can make full use of this information to make action decisions. This information does not include the agent's visual data because visual data depends on the map and is massive in volume; transmitting it over the network would waste bandwidth and affect performance. Therefore, visual data is dynamically rendered in the AIServer, specifically as described in step 202.

[0059] 202. Based on the game scene data and the behavior observation data, determine the visual data of each agent, wherein the visual data is used to reflect the object information within the visual range of the agent in the target game scene;

[0060] It should be noted that due to the free-building nature of sandbox games, the world structure changes frequently. Therefore, in order to reduce the pressure of data transmission, the aforementioned game data and behavioral observation data do not include the agent's visual data. Visual data is used to reflect the information of objects within the agent's visual range in the target game scene. Therefore, in order to model the agent, it is necessary to obtain the agent's visual data. The agent's visual data is then dynamically generated on the AI ​​server side and rendered in the AI ​​server using game scene data and behavioral observation data to obtain the agent's visual data.

[0061] For example, the visual data of an intelligent agent refers to information about objects near the agent, including but not limited to information about nearby items and characters. The information about nearby items includes, but is not limited to, item ID, type, and location, which is equivalent to the intelligent agent player seeing available items or obstacles on the ground through the screen. The information about nearby characters includes, but is not limited to, information about nearby characters, including enemies, teammates, and NPCs. Its content is similar to the player information, but it does not nest the nearby information.

[0062] 203. Based on the visual data and the action decision model, determine the target action instruction for each agent. The target action instruction is used to control the agent to perform actions that conform to the game survival rules. The action decision model is a model trained using reinforcement learning.

[0063] Furthermore, after obtaining the aforementioned visual data, decisions can be made regarding the behavior of each agent based on the data surrounding each agent. Specifically, based on the visual data and the action decision model, a target action instruction is determined for each agent. This target action instruction controls the agent to perform actions that conform to the game's survival rules. The action decision model is a deep learning model trained using reinforcement learning.

[0064] Furthermore, the agent, having acquired visual data, needs to make action decisions. Just as human players in games know to run when danger approaches or pick up treasure when they see it, every action decision ultimately aims to gain a reward. Similarly, the agent model trained through reinforcement learning executes actions to obtain rewards, and may even sacrifice short-term gains for long-term, substantial benefits. That is, action instructions not only include the actions the agent needs to perform but also the corresponding reward data. For example, agent actions include, but are not limited to, movement, perspective, and behavior. Movement actions include, but are not limited to, walking, running, jumping, and sneaking; perspective actions include, but are not limited to, looking up and looking down; and behavior actions include, but are not limited to, using items, picking up items, equipping gear, crafting items, and switching items. These behavior instructions can basically cover a fairly complete gameplay experience. Taking Minecraft as an example, in the early stages of the game, players need to scratch trees to obtain items; the hand itself is an item. By consistently applying the "use item" command, the agent can perform the action of scratching trees and obtain the item wood blocks (reward). Similarly, by adjusting your position and view to aim at the target location, switching your inventory to "wooden blocks," and finally applying the item command, you can place the wooden blocks on the ground. Placing multiple wooden blocks will build a house. Weapons in the game, such as bows and swords, can also be used—that is, attacked—using item commands.

[0065] 204. Send the target action instruction to the client corresponding to each agent, and return to execute the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all agents pushed by the preset host client in the target game scene, until the action decision model converges, then the modeling of the agents in the target game scene is completed.

[0066] After obtaining the target action instruction, it can be sent to the client corresponding to the agent, causing the agent in the target game scene to execute the corresponding target action instruction. After the agent performs the action, the process can return to the steps of obtaining the game data of the target game scene from the AI ​​server, as well as the behavior observation data of all agents pushed by the preset host client in the target game scene, to verify the execution effect of the agent's action instruction. For example, by using the agent's reward data, it can be determined whether the agent's survival status conforms to the game survival rules, thus determining the execution effect of the action instruction. If the error of the execution effect is within the allowable error range, it means that the agent has performed the corresponding action well according to the action instruction. At this time, the action decision model converges, and the modeling of the agent in the target game scene is completed. If the error is not within the allowable error range, it means that the agent has not performed the action well according to the action instruction, and the instruction output by the model does not yet have the ability to conform to the game survival rules. Therefore, it is necessary to continue modeling the agent, that is, return to step 201.

[0067] For example, the AIServer uses a model to predict the data returned by the agent (client), determining which actions the agent should perform, and then sends these actions to the game client for execution. The question then becomes how well the agent performs. This requires monitoring the agent's performance on the game client side. For instance, does the agent collect items efficiently? When heading to a location, does it take a detour or walk directly there? These observations are essential for validating and debugging the model, determining whether it has converged and whether the agent possesses autonomous decision-making capabilities.

[0068] This invention provides a method for modeling intelligent agents. The method is applied to an AI server, with at least one intelligent agent and a one-to-one correspondence between intelligent agents and clients. The method includes: acquiring game data of a target game scene from the AI ​​server, and behavioral observation data of all intelligent agents pushed by a preset host client in the target game scene. The game data reflects information about players and the scene in the target game scene, and includes at least game scene data; determining visual data for each intelligent agent based on the game scene data and behavioral observation data. The visual data reflects information about objects within the visual range of the intelligent agent in the target game scene; determining target action instructions for each intelligent agent based on the visual data and an action decision model. The target action instructions control the intelligent agent to perform actions that conform to the game's survival rules. The action decision model is a model trained using reinforcement learning; sending the target action instructions to the client corresponding to each intelligent agent, and returning to the steps of acquiring game data of the target game scene from the AI ​​server and behavioral observation data of all intelligent agents pushed by a preset host client in the target game scene, until the action decision model converges, thus completing the modeling of the intelligent agents in the target game scene. The above approach enables the visual data of agents in open worlds to be dynamically rendered and generated on AI servers. This reduces the bandwidth wastage and performance impact caused by transmitting agent visual data over the network due to the complexity of freely constructed scenes in open worlds. It meets the performance requirements of online access while also controlling the cost of offline training. Furthermore, the action decision-making model trained using reinforcement learning can make action decisions for the agent, resulting in more human-like behavior.

[0069] Please see Figure 3 , Figure 3 This is another flowchart of a modeling method for an intelligent agent according to an embodiment of the present invention, as follows: Figure 3 The method is applied to an AI server, and the number of intelligent agents is at least one. Each intelligent agent corresponds one-to-one with a client. The method includes the following steps:

[0070] 301. Obtain game data of the target game scene in the AI ​​server, as well as behavioral observation data of all intelligent agents pushed by the host client in the target game scene. The game data is used to reflect the information of the players and the scene in the target game scene, and the game data includes at least game scene data.

[0071] It should be noted that step 301 and Figure 2 Step 201 shown is similar in content, and to avoid repetition, it will not be repeated here. Please refer to the previous section for details. Figure 2 The content of step 201 is shown.

[0072] In one feasible implementation, a pre-defined host client is required before step 301, and therefore, the process before step 301 also includes:

[0073] A1. Receive the communication establishment request sent by the client;

[0074] A2. In response to the communication request, establish a communication connection between the client and the AI ​​server;

[0075] A3. After successfully establishing a communication connection between the client and the AI ​​server, if the client is the first client to successfully establish a communication connection with the AI ​​server, then the client is determined to be the target client; if the client is not the first client to successfully establish a communication connection with the AI ​​server, then the client is determined to be the passenger client.

[0076] It should be noted that the host client refers to the first client to establish a communication connection with the AI ​​server. Therefore, when establishing communication between a client and the AI ​​server, the AI ​​server can determine which client is the host client. Specifically, it receives the communication request sent by the client in step A1, and responds to the communication request in step A2 to establish a communication connection between the two. After the communication is established, step A3 determines whether the client is the first client to successfully establish a communication connection, thereby identifying the first host client that has successfully established a communication connection with the AI ​​server.

[0077] In one feasible implementation, the method further includes:

[0078] B1. Send a room creation request to the host client and receive the room creation result returned by the host client;

[0079] B2. When the room creation result is successful, the room is initialized and configured according to the room identifier and training requirements in the room creation result to obtain the target game scene. The training requirements include the game scene, the agent training scale, and the agent's ability attributes.

[0080] It's important to note that in this implementation, after obtaining the aforementioned host client, a room creation request can be sent to the host client. This allows the host client to perform room creation operations on the game cloud server. The game cloud server then returns the room creation result to the host client, which can then send this result to the AI ​​server. The room creation result includes, but is not limited to, room information such as a room identifier. The AI ​​server can then use this identifier to perform initial configuration processing on the room. This initial configuration processing is to configure the room parameters to meet training requirements, including, but not limited to, the game scene, the training scale of the agent, and the agent's ability attributes. Specifically, game data refers to the information acquired by the agent in the game environment, which can only be obtained on the client side. Similarly, the actions given by the AI ​​Server to the agent also need to be executed on the client side. For example, if the agent observes that there is an apple 5 meters away, then when training the agent's ability to collect items, the agent needs to perform the action of walking to that location 5 meters away and picking up the apple. In a game, this might correspond to controlling a character to move towards the target, stop, and send the command to pick up the item. After picking it up, the agent's next status information will show that its backpack has an additional apple. All intelligent agents need to perform actions to achieve a certain goal in the game environment (displayed through state) in order to train the model. This goal can be numerous, such as collecting items, fighting, building, etc. These scenarios need to be configured through initialization, allowing the intelligent agent to encounter these real-world scenarios and train the model. The training requirements can be met through initialization configuration.

[0081] In one feasible implementation, the initial configuration process for the room, based on the room identifier and training requirements in the room creation result, further includes:

[0082] C1. Send a room join request to the passenger aircraft client, the room join request including the room identifier;

[0083] C2. Receive the room joining result returned by the passenger aircraft client;

[0084] C3. Count the total number of passenger clients that successfully joined the room in the room joining results;

[0085] C4. When the total number is not less than the training scale of the intelligent agent, the game start command sent by the host client is received, and the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all intelligent agents pushed by the host client in the target game scene are executed.

[0086] It should be noted that after the room initialization is complete, other clients need to be invited to join the room and play the game. Therefore, a join request is sent to the passenger client through step C1, so that the passenger client's join request includes the game room corresponding to the room identifier. In addition, the AI ​​server can also set the agent training scale, that is, how many agents to train. The AIServer can manage multiple agents at the same time, and will make decisions based on the state feedback of each agent using the model trained by reinforcement learning, and return the action instructions to be executed. In order to ensure the training scale, the total number of passenger clients who successfully joined the room can be obtained by counting the received join results through C2 and C3. Since one client corresponds to one agent, the total number of clients represents the total number of agents. Therefore, when the total number is not less than the agent training scale, agent training can start. Through C4, the host client initiates the game start command, starts agent modeling, and executes step 301.

[0087] 302. Determine the visual range of each intelligent agent based on the preset boundary range, the position data, and the orientation data;

[0088] The behavioral observation data includes the agent's position data and orientation data. The visual range of each agent can be determined by using a preset boundary range, the position data, and the orientation data. The preset boundary range is the range within a preset radius centered on the agent's position data. The specific radius is related to the game settings. Furthermore, the orientation data is combined to obtain the agent's visual range. For example, if the orientation data indicates that the agent is facing east and the radius is 50m, then the visual range is the area within 50m east of the agent's position data.

[0089] 303. Perform scene rendering processing based on the visual range, the static map data, and the dynamic construction data to obtain depth map data within the visual range of each agent. Use the depth map data as the visual data of the agent. The visual data is used to reflect the object information within the visual range of the agent in the target game scene.

[0090] Furthermore, the game scene data includes static map data and dynamic construction data of the target game scene. Scene rendering can then be performed based on the visual range, the static map data, and the dynamic construction data to obtain depth map data within the visual range of each agent. This depth map data is then used as the agent's visual data. Specifically, taking sandbox games as an example, many sandbox games use blocks to build the game world. The positional information of these blocks can describe the entire world's terrain. This information can be converted into depth map information to obtain object information within the visual range, allowing the agent to know whether obstacles have appeared in front of it. This information is bound to the map and can be pre-loaded or generated and modified as needed. The depth map obtained by rendering on the AI ​​server side is a black and white image; the depth of the black areas indicates the distance of obstacles at corresponding locations in the field of vision.

[0091] 304. Based on the visual data and the action decision model, determine the target action instruction for each agent. The target action instruction is used to control the agent to perform actions that conform to the game survival rules. The action decision model is a model trained using reinforcement learning.

[0092] It should be noted that step 304 and Figure 2 The content of step 203 shown is similar, and to avoid repetition, it will not be repeated here. Please refer to the previous text for details. Figure 2 The content of step 203 shown.

[0093] In one feasible implementation, the visual data includes at least the type of obstacle and the distance between the obstacle and the agent. Then, based on the visual data and the action decision model, the target action instruction for each agent is determined, including: inputting the type of obstacle and the distance between the obstacle and the agent into the action decision model to determine the target action instruction for each agent.

[0094] It should be noted that the visual data described above can be used to determine the type of obstacle and the distance between the obstacle and the agent. The obstacle type can be a static obstacle obtained from static map data, such as rocks in map landscaping data, or a dynamic obstacle obtained from dynamic construction data, such as freely constructed bricks. The obstacle type and the distance between the obstacle and the agent can be input into the action decision model, which outputs the target action command for each agent. It is understood that different agents may have different visual data, resulting in different action commands. This allows for teamwork by having different agents execute different action commands, further enhancing the anthropomorphism of the agents.

[0095] Furthermore, static obstacles refer to fixed obstacle information on a game map, typically represented in games using .obj files. These files are usually described using points, lines, edges, and other information to depict the polygonal data of the entire obstacle network. These .obj files are generally stored internally within the AI ​​server. Knowing the agent's position and orientation allows for the timely rendering of obstacles within what the agent sees, represented by a depth map. Dynamic obstacles, on the other hand, are not part of the map .obj files but are obtained through synchronized data. In sandbox games, characters can often place blocks. These blocks can be identified using the aforementioned "nearby item information," indicating their location. The polygons for each block can be pre-prepared and rendered only as needed. Ultimately, by combining dynamic and static obstacles, the agent obtains complete rendering information.

[0096] In one feasible implementation, the game data further includes agent reward data and agent attribute data. Then, determining the target action instruction for each agent based on the visual data and the action decision model further includes:

[0097] The visual data, game scene data, character reward data, and character attribute data are input into the action decision model to determine the target action instruction for each agent.

[0098] Furthermore, in sandbox games like Minecraft, player-controlled characters often possess a range of survival attributes. Besides common stats like health, these include hunger, fatigue, and oxygen levels. Low hunger necessitates eating to prevent health depletion and death. High fatigue affects character behavior, and zero oxygen in water simulates drowning, leading to health loss and death. These are all survival factors, requiring agents to consider more survival rules. To better obtain target action commands for the agent, more feature data can be input into the model. Specifically, visual data, game scene data, character reward data, and character attribute data can be input into the action decision model to determine the target action commands for each agent.

[0099] 305. Send the target action instruction to the client corresponding to each of the intelligent agents, and return to execute the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all intelligent agents pushed by the preset host client in the target game scene, until the action decision model converges, then the modeling of the intelligent agents in the target game scene is completed.

[0100] It should be noted that step 305 and Figure 2 The content of step 204 shown is similar, and to avoid repetition, it will not be repeated here. For details, please refer to the content of step 204 above.

[0101] To better understand the modeling process of this application, please refer to [link / reference]. Figure 4 , Figure 4 This is an interactive flowchart of the training process of a modeling method for an intelligent agent according to an embodiment of the present invention. Figure 4 This includes host clients, AIServer, and passenger clients. Figure 4 To illustrate the interaction process between the host client, AIServer, and passenger client during training, the training process is specifically divided into four stages: (1) game start stage, (2) game progress stage, (3) game end stage, and (4) game entry / exit stage.

[0102] (1) Game Start Phase: In this phase, the game client initiates a connection to the AIServer using the WebSocket protocol, which is essentially a long-connection, two-way communication protocol based on HTTP. A game involves multiple game clients as participants, with the first client acting as the host and the others as clients. After establishing a connection, each client periodically synchronizes its heartbeat with the AIServer to ensure service connectivity.

[0103] After the host client connects, the AIServer requests room creation and sandbox world initialization, such as selecting a map, number of players, AI positions, game mode, etc. After room creation, a waiting phase begins, awaiting client connections. For offline training scenarios (modeling), the number of clients is determined by the AI ​​side, so a corresponding number of clients must connect. Each joining client receives a configuration command from the AIServer to join the room. For online scenarios, the game's backend cloud service coordinates the number of clients connecting. Once the room is full, the host client will proactively notify the game to start and proceed to the next phase.

[0104] (2) Game Progression Phase: This phase constitutes the main part of the entire process and is logically relatively simple. The host client pushes the AI's state information throughout the game, and the model in the AIServer makes decisions and pushes corresponding action decisions. This achieves state synchronization between the server and the client, and sends action instructions to the client to enable the agent to act.

[0105] (3) Game End Phase: There are two scenarios in this phase. The first is when the game reaches the end condition. In this case, the host client will actively push the game end information, and the AIServer will decide whether to restart the game based on the training requirements. If restarting is required, it will return to the game start phase for initial configuration. The second scenario is when the AIServer actively determines that the game needs to be restarted for training on more scenarios. In this case, it will actively send a request to restart the game and enter the game start phase in the same way.

[0106] The scenarios can be divided into two aspects: the same map or different maps. For the same map, a typical game map contains many scenes, such as towns, forests, and lakes. The game tasks encountered in each scene are different. For example, in survival mode, in a forest, one might need to chop down trees and hunt wild boars, while at the seaside, one might need to fish, and so on. Since in actual game scenarios, an agent might be anywhere on the map and encounter any element, during training, the agent needs to receive sample data from these scenes and be trained in a way that leads to specific rewards. For example, in a forest, chopping down trees to obtain wood and crafting tools yields a reward, as obtaining tools is a useful action. Similarly, hunting wild boars for food also yields a reward. In this way, the trained AI will learn to collect food and craft tools, thus understanding survival in a sense.

[0107] (4) Game entry and exit phase: This phase marks the official end of the game, during which the game client and AIServer disconnect one after another.

[0108] This invention provides a method for modeling intelligent agents that addresses the anthropomorphic requirements and performance standards of 3D sandbox games. It trains the intelligent agent model using reinforcement learning, enabling it to adapt to the complex scenes and diverse gameplay of sandbox games, exhibiting more human-like behavior. This method meets the performance requirements of online integration while also controlling the cost of offline training. Specifically, it reuses offline training and online training methods. Figure 4 The access framework shown reduces development effort. There are two key differences between offline and online training: offline training requires model iteration, while online training uses a fixed model to provide decisions; secondly, offline training allows the AIServer to freely specify the training scenario, while online training is based on the game's gameplay logic. Aside from this difference, the rest of the access process is the same as the online process. For each request, the game client sends a request, and the AIServer responds. The performance benefit of reusing the same process is that, online, the AIServer doesn't need to actively push messages to the game client, avoiding unnecessary connection establishment and logic processing. Furthermore, the synchronization decision for each request rests with the game client, allowing for control over online latency and other factors.

[0109] Please see Figure 5 , Figure 5 This is a structural block diagram of a modeling device for an intelligent agent according to an embodiment of the present invention, such as... Figure 5 The device is applied to an AI server, and the number of intelligent agents is at least one, with each intelligent agent corresponding to a client. The device includes:

[0110] Data acquisition module 501: used to acquire game data of the target game scene in the AI ​​server, and behavioral observation data of all intelligent agents pushed by the preset host client in the target game scene. The game data is used to reflect the information of the players and the scene in the target game scene, and the game data includes at least game scene data.

[0111] Visual determination module 502: used to determine the visual data of each intelligent agent based on the game scene data and the behavior observation data, wherein the visual data is used to reflect the object information within the visual range of the intelligent agent in the target game scene;

[0112] Action decision module 503: used to determine the target action instruction for each agent based on the visual data and the action decision model. The target action instruction is used to control the agent to perform actions that conform to the game survival rules. The action decision model is a model trained using reinforcement learning.

[0113] Action execution module 504: Used to send the target action instruction to the client corresponding to each of the intelligent agents, and return to execute the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all intelligent agents pushed by the preset host client in the target game scene, until the action decision model converges, and then the modeling of the intelligent agents in the target game scene is completed.

[0114] It should be noted that, Figure 5 The function of each module in the device shown is as follows: Figure 2 The steps in the method shown are similar, and will not be repeated here to avoid repetition. Please refer to the previous section for details. Figure 2 The steps in the method shown are as follows.

[0115] This invention provides a modeling device for intelligent agents. The device is applied to an AI server, and the number of intelligent agents is at least one. Each intelligent agent corresponds to a client. The device includes: a data acquisition module, used to acquire game data of a target game scene from the AI ​​server, and behavioral observation data of all intelligent agents pushed by a preset host client in the target game scene. The game data reflects information about players and the scene in the target game scene, and includes at least game scene data; a vision determination module, used to determine the visual data of each intelligent agent based on the game scene data and behavioral observation data. The visual data reflects information about objects within the visual range of the intelligent agent in the target game scene; an action decision module, used to determine the target action command for each intelligent agent based on the visual data and an action decision model. The target action command controls the intelligent agent to perform actions that conform to the game's survival rules. The action decision model is a model trained using reinforcement learning; and an action execution module, used to send the target action command to the client corresponding to each intelligent agent and return to execute the steps of acquiring game data of the target game scene from the AI ​​server and behavioral observation data of all intelligent agents pushed by a preset host client in the target game scene, until the action decision model converges, thus completing the modeling of the intelligent agents in the target game scene. The above approach enables the visual data of agents in open worlds to be dynamically rendered and generated on AI servers. This reduces the bandwidth wastage and performance impact caused by transmitting agent visual data over the network due to the complexity of freely constructed scenes in open worlds. It meets the performance requirements of online access while also controlling the cost of offline training. Furthermore, the action decision-making model trained using reinforcement learning can make action decisions for the agent, resulting in more human-like behavior.

[0116] Figure 6 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned methods. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned methods. Those skilled in the art will understand that… Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform actions such as... Figure 2 or Figure 3 The steps of the method shown.

[0118] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following actions: Figure 2 or Figure 3 The steps of the method shown.

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method of modeling an agent, the method comprising: The method is applied to an AI server, wherein there is at least one intelligent agent, and each intelligent agent corresponds one-to-one with a client. The method includes: The game data of the target game scene in the AI ​​server is obtained, as well as the behavior observation data of all intelligent agents pushed by the host client in the target game scene. The game data is used to reflect the information of the players and the scene in the target game scene, and the game data includes at least game scene data. Based on the game scene data and the behavior observation data, visual data for each agent is determined. The visual data reflects information about objects within the visual range of the agent in the target game scene. The visual data is dynamically rendered and generated on the AI ​​server side based on the game scene data and the behavior observation data. Based on the visual data and the action decision model, a target action instruction is determined for each agent. The target action instruction is used to control the agent to perform actions that conform to the game's survival rules. The action decision model is a model trained using reinforcement learning. The target action command is sent to the client corresponding to each agent, and the process of obtaining game data of the target game scene in the AI ​​server and behavioral observation data of all agents pushed by the preset host client in the target game scene is returned until the action decision model converges, thus completing the modeling of agents in the target game scene.

2. The method of claim 1, wherein, The behavioral observation data includes the agent's position and orientation data, and the game scene data includes static map data and dynamic construction data of the target game scene. Therefore, determining the visual data of each agent based on the game scene data and the behavioral observation data includes: The visual range of each intelligent agent is determined based on the preset boundary range, the location data, and the orientation data; Scene rendering is performed based on the visual range, the static map data, and the dynamic construction data to obtain depth map data within the visual range of each agent, and the depth map data is used as the visual data of the agent.

3. The method of claim 2, wherein, The visual data includes at least the type of obstacle and the distance between the obstacle and the agent. Based on the visual data and the action decision model, the target action command for each agent is determined, including: The type of obstacle and the distance between the obstacle and the agent are input into the action decision model to determine the target action instruction for each agent.

4. The method of claim 3, wherein, The game data also includes agent reward data and agent attribute data. Therefore, determining the target action command for each agent based on the visual data and the action decision model further includes: The visual data, game scene data, character reward data, and character attribute data are input into the action decision model to determine the target action instruction for each agent.

5. The method of claim 1, wherein, The process of acquiring game data of the target game scene from the AI ​​server, and behavioral observation data of all intelligent agents pushed by the preset host client in the target game scene, also includes the following: Receive a communication establishment request sent by the client; In response to the communication request, a communication connection is established between the client and the AI ​​server; After successfully establishing a communication connection between the client and the AI ​​server, if the client is the first client to successfully establish a communication connection with the AI ​​server, then the client is determined to be the host client; if the client is not the first client to successfully establish a communication connection with the AI ​​server, then the client is determined to be the passenger client.

6. The method of claim 5, wherein, The method further includes: Send a room creation request to the host client and receive the room creation result returned by the host client; When the room creation result is successful, the room is initialized and configured according to the room identifier and training requirements in the room creation result to obtain the target game scene. The training requirements include the game scene, agent training scale, and agent capability attributes.

7. The method of claim 6, wherein, The initial configuration process for the room, based on the room identifier and training requirements in the room creation result, further includes: Send a room join request to the passenger aircraft client, the room join request including the room identifier; Receive the room joining result returned by the passenger aircraft client; Count the total number of client machines that successfully joined the room in the room joining results; When the total number is not less than the training scale of the agent, the game start command sent by the host client is received, and the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all agents pushed by the host client in the target game scene are executed.

8. An apparatus for modeling an agent, the apparatus comprising: The device is applied to an AI server, and the number of intelligent agents is at least one, with each intelligent agent corresponding to a client. The device includes: Data acquisition module: used to acquire game data of the target game scene in the AI ​​server, as well as behavioral observation data of all intelligent agents pushed by the preset host client in the target game scene. The game data is used to reflect the information of players and scene in the target game scene, and the game data includes at least game scene data. Visual determination module: used to determine the visual data of each agent based on the game scene data and the behavior observation data. The visual data is used to reflect the object information within the visual range of the agent in the target game scene. The visual data is dynamically rendered and generated on the AI ​​server side based on the game scene data and the behavior observation data. Action Decision Module: Used to determine the target action instruction for each agent based on the visual data and the action decision model. The target action instruction is used to control the agent to perform actions that conform to the game survival rules. The action decision model is a model trained using reinforcement learning. Action execution module: Used to send the target action instruction to the client corresponding to each of the intelligent agents, and return to execute the steps of obtaining the game data of the target game scene in the AI ​​server, and the behavior observation data of all intelligent agents pushed by the preset host client in the target game scene, until the action decision model converges, thus completing the modeling of the intelligent agents in the target game scene.

9. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7. 10.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images