Intelligent agent model training method, intelligent agent control method, intelligent agent model training device, intelligent agent control device and medium

Abstract

The invention provides an agent model training method and device, an agent control method and device and a medium, and the training method comprises the steps: obtaining the current state information of an agent in a game scene at the current moment, and processing the current state information according to an initial decision model of the agent, the next behavior decision of the agent at the next moment in the game scene is predicted, the next state information of the agent when the agent executes the next behavior decision at the next moment is obtained, the next state information comprises the interaction state of the agent and other agents, according to the interaction state, the actual reward feedback when the agent executes the next behavior decision is obtained, and the actual reward feedback is sent to the agent when the agent executes the next behavior decision. And training the initial decision model according to the actual reward feedback to obtain a target decision model of the intelligent agent. And the decision model is trained in combination with reward feedback in reinforcement learning, and the decision model of the agent is automatically generated, so that the authenticity and diversity of agent behaviors in a game group environment are improved.

Description

Technical Field

[0001] The present application relates to the field of game technology, and more specifically, to an intelligent agent model training method, an intelligent agent control method, a device and a medium. Background Art

[0002] In game development, swarm intelligence technology is often used to create groups of non-player characters (NPCs) with complex interactive behaviors. These NPC groups can make decisions and react autonomously based on the game environment and the player's behavior, thus providing players with a more realistic and immersive gaming experience.

[0003] Currently, the implementation of swarm intelligence mainly relies on predefined rules, which guide the behavior of NPCs in specific situations. Among them, behavior tree is a common rule representation method, which defines the possible behavior sequence of NPCs through a tree structure.

[0004] However, rule-based approaches may lead to NPC behavior patterns being too rigid, making it difficult to exhibit truly natural and diverse behaviors. Summary of the invention

[0005] In view of this, the embodiments of the present application provide an intelligent agent model training method, an intelligent agent control method, a device and a medium to solve the problem that NPC behavior patterns are too stereotyped and difficult to show truly natural and diverse behaviors.

[0006] In a first aspect, an embodiment of the present application provides an agent model training method, comprising:

[0007] Get the current state information of the agent in the game scene at the current moment;

[0008] Processing the current state information according to the initial decision model of the agent to predict the next behavior decision of the agent at the next moment;

[0009] Acquire next state information of the agent when executing the next behavior decision at the next moment, the next state information including: interaction state of the agent and other agents;

[0010] According to the interaction state, obtaining actual reward feedback when the agent executes the next behavior decision;

[0011] The initial decision model is trained according to the actual reward feedback to obtain the target decision model of the agent.

[0012] In a second aspect, an embodiment of the present application provides an intelligent agent control method, comprising:

[0013] Get the state information of the agent in the game scene at the target time;

[0014] Processing the state information at the target moment according to the target decision model of the agent to obtain the next behavior decision of the agent at the next moment after the target moment;

[0015] According to the next behavior decision, the agent is controlled to perform a corresponding behavior action at the next moment after the target moment.

[0016] In a third aspect, the embodiment of the present application further provides an intelligent agent model training device, comprising:

[0017] The acquisition module is used to obtain the current state information of the agent in the game scene at the current moment;

[0018] A processing module, used to process the current state information according to the initial decision model of the intelligent agent, and predict the next behavior decision of the intelligent agent at the next moment;

[0019] The acquisition module is further used to acquire the next state information of the agent when executing the next behavior decision at the next moment, and the next state information includes: the interaction state of the agent and other agents;

[0020] The acquisition module is further used to acquire actual reward feedback when the agent executes the next behavior decision according to the interaction state;

[0021] The processing module is also used to train the initial decision model according to the actual reward feedback to obtain the target decision model of the agent.

[0022] In a fourth aspect, an embodiment of the present application further provides an intelligent body control device, comprising:

[0023] The acquisition module is used to obtain the state information of the agent in the game scene at the target time;

[0024] A processing module, used for processing the state information of the target moment according to the intelligent agent control model to obtain the next behavior decision of the intelligent agent at the next moment after the target moment;

[0025] The processing module is used to control the agent to perform corresponding behavioral actions in the game scene according to the next behavior decision.

[0026] In the fifth aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to execute any one of the methods described in the first aspect.

[0027] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the first aspects is executed.

[0028] The present application provides an agent model training method, an agent control method, a device and a medium, wherein the training method includes: obtaining the current state information of the agent in the game scene at the current moment, processing the current state information according to the initial decision model of the agent, predicting the next behavior decision of the agent in the game scene at the next moment, obtaining the next state information of the agent in the game scene when executing the next behavior decision at the next moment, and the next state information includes: the interaction state of the agent and other agents, obtaining the actual reward feedback when the agent executes the next behavior decision according to the interaction state, training the initial decision model according to the actual reward feedback, and obtaining the target decision model of the agent. The decision model is trained in combination with the reward feedback in reinforcement learning, and the decision model of the agent is automatically generated, thereby improving the authenticity and diversity of the agent behavior in the game group environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 1 ;

[0031] Figure 2 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 2 ;

[0032] Figure 3 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 3 ;

[0033] Figure 4Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 4 ;

[0034] Figure 5 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 5 ;

[0035] Figure 6 A schematic diagram of a flow chart of an intelligent agent control method provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the agent model training process provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of the structure of an intelligent agent model training device provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of the structure of an intelligent body control device provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0041] First, the professional terms designed in this application are explained:

[0042] Language Model (LM): An artificial intelligence model used to predict the probability of the next word or character, usually used in natural language processing tasks.

[0043] Reinforcement Learning (RL): A machine learning approach in which an agent learns optimal behavior policies by interacting with its environment.

[0044] Intelligent Group Training: It combines the theories and methods of artificial intelligence, machine learning and group intelligence to train groups composed of multiple intelligent agents.

[0045] Game Non-Player Character (NPC): A character in a video game that is controlled by a computer program. They are not directly operated by the player, but perform preset behaviors and tasks in the game. NPCs are usually used to provide storylines, tasks, dialogues, interactions, and other game content to enrich the game world and enhance the player's gaming experience.

[0046] Early implementations of swarm intelligence mainly relied on predefined rules that guided the behavior of NPCs in specific situations. Behavior trees are a common rule representation method that defines the possible behavior sequences of NPCs through a tree structure. This method is simple and intuitive, but it is difficult to handle complex interactions and dynamic environments.

[0047] However, rule-based methods require developers to have a deep understanding of game design, psychology, sociology, and artificial intelligence. In addition, they also need to be able to transform abstract concepts into specific rules and parameters. This high threshold may limit developers' innovation and ability to implement complex behaviors. In addition, it may cause NPC behavior patterns to be too stereotyped, making it difficult to show truly natural and diverse behaviors.

[0048] Based on this, the present application provides a user-friendly tool and framework developed in combination with a reinforcement learning framework. It only requires initializing model parameters so that non-expert developers can easily create and adjust intelligent agent behaviors, and automatically generate richer and more realistic intelligent agent behaviors, thereby improving the authenticity and diversity of intelligent agent behaviors.

[0049] Figure 1 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 1 The execution subject of this embodiment may be an electronic device.

[0050] like Figure 1 As shown, the method may include:

[0051] S101, obtaining the current state information of the agent in the game scene at the current moment.

[0052] The agent is any character controlled by a computer program in the game scene, such as an NPC. The current state information is the state information of the agent in the game scene at the current moment. The current state information may include the position, environment, and behavior of the agent in the game scene at the current moment. The behavior may be, for example, sleeping, cooking, chatting, and exercising.

[0053] S102: Process the current state information according to the initial decision model of the intelligent agent, and predict the next behavior decision of the intelligent agent at the next moment.

[0054] The initial decision model can be a decision model in a reinforcement learning framework. The current state information is input into the initial decision model of the agent to predict the next behavior decision of the agent at the next moment. The next behavior decision is the behavior decision performed by the agent at the next moment in the game scene, which is used to indicate the behavior action performed by the agent at the next moment, such as sleeping, cooking, chatting, exercising, etc., wherein the next moment is the next moment of the current moment, for example, if the current moment is moment t-1, then the next moment is moment t.

[0055] S103, obtaining the next state information of the intelligent agent when it performs the next behavior decision at the next moment.

[0056] Control the agent to execute the next behavior decision at the next moment in the game scene, and obtain the next state information of the agent when the agent executes the next behavior decision at the next moment in the game scene, wherein the next state information is the state information of the agent in the game scene at the next moment, and the next state information includes: the interaction state between the agent and other agents, and other agents are other characters controlled by computer programs in the game scene except the agent, such as NPCs.

[0057] The interaction state is used to indicate the interaction events between an agent and other agents, such as communication dialogue events, collaborative operation events (such as moving virtual items, attacking enemies, etc.).

[0058] S104. According to the interaction state, obtain the actual reward feedback when the agent executes the next behavior decision.

[0059] According to the interaction status of the agent and other agents, the actual reward feedback of the agent when executing the next behavior decision is obtained. The actual reward feedback is used to indicate the degree of interaction between the agent and other agents when executing the next behavior decision. The more interaction events, the higher the degree of interaction, and the more actual reward feedback; the fewer interaction events, the lower the degree of interaction, and the less actual reward feedback.

[0060] For example, if an interaction event is completed well, the actual reward feedback is 1, if the interaction event is not completed, the actual reward feedback is -1, and if an interaction event is barely completed, the actual reward feedback is 0.

[0061] In an optional embodiment, the next state information may also include: an event reward state, where the event reward state is used to indicate a preset reward event performed by the agent. The preset reward event may include an unexpected event or a dramatic event. An additional point reward (such as an additional point reward between 0 and 1) may be set for the preset reward event to enhance the interest of the agent's actions.

[0062] Among them, the preset reward events can be, for example, whether or not to chat while eating, or to enter a working state (such as cleaning an inn).

[0063] S105. According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the intelligent agent.

[0064] According to the actual reward feedback, the model parameters of the initial decision model of the intelligent agent are updated to train the initial decision model until the preset stopping condition is met. The decision model obtained when the preset stopping condition is met is used as the target decision model of the intelligent agent.

[0065] It is understandable that, in the process of reinforcement learning, the next state information of the agent after the execution of the next behavior decision can also be collected. For example, the state when executing the next behavior decision (such as eating) is eating, and the state after the execution of the next behavior decision is sleeping (that is, the state of the next moment of the next moment is sleeping). Compared with eating, sleeping is the state of the agent after executing the next behavior decision. Then, according to the next state information of the agent after the execution of the next behavior decision, the above steps S101-S105 are repeatedly executed, and the state information of the agent at each moment is continuously collected and the model parameters of the initial decision model are updated until the preset stop condition is met.

[0066] In this embodiment, a reinforcement learning framework is combined to develop user-friendly tools and frameworks that only require initialization of model parameters, so that non-expert developers can easily create and adjust agent behaviors, and automatically generate richer and more realistic agent behaviors, thereby improving the authenticity and diversity of agent behaviors in game group environments.

[0067] In an optional implementation, the above step S104, obtaining actual reward feedback when the agent performs the next behavior decision according to the interaction state, may include:

[0068] According to the interaction state and the reward feedback rule information based on the natural language description, the first preset large language model is adopted to obtain the actual reward feedback.

[0069] Among them, the first preset large language model can be, for example, a GPT4 model, which provides basic cognitive capabilities and has fixed model parameters.

[0070] The reward feedback rule information based on natural language description is used to indicate the reward rules corresponding to the interaction events. For example, when an intelligent agent has at least one conversation with other intelligent agents every day, a feedback of +1 is artificially set. If an intelligent agent has no communication with other intelligent agents at all, a feedback of -1 is given. This reward feedback reflects the hope that the intelligent agents will communicate more so as to meet the required intelligence requirements.

[0071] The reward feedback rule information can be expressed as "please encourage more communication between agents and design the reward mechanism accordingly. If an agent has no communication with other agents at all, then give -1 feedback."

[0072] The interaction state and reward feedback rule information are used as input of the first preset large language model, and the output is the actual reward feedback corresponding to the interaction state.

[0073] It is worth noting that when the current status information includes the event reward status, the reward feedback rules can also include feedback rules corresponding to the reward event. For example, when the intelligent agent performs an emergency event, a feedback of +1 is manually set to enhance the fun of the intelligent agent's actions. The specific settings can be based on actual conditions, and this embodiment does not specifically limit this.

[0074] Figure 2 Schematic diagram of the process of the intelligent model training method provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, in an optional implementation, the method may further include:

[0075] S201. Process the interaction state according to the initial value model of the intelligent agent to obtain the predicted reward feedback of the intelligent agent at the next moment.

[0076] The initial value model is a value model in the reinforcement learning framework. The interaction state in the next state information is input into the initial value model of the agent to obtain the predicted reward feedback of the agent at the next moment, where the predicted reward feedback is the reward feedback of the agent at the next moment predicted by the initial value model, which is used to characterize the predicted intelligence level of the agent.

[0077] In an optional implementation, the above step S105, training the initial decision model according to the actual reward feedback to obtain the target decision model of the agent, may include:

[0078] S202: Calculate the advantage function based on the actual reward feedback and the predicted reward feedback.

[0079] S203: Train the initial decision model according to the advantage function to obtain the target decision model.

[0080] The advantage function can be calculated based on the actual reward feedback and the predicted reward feedback. The advantage function is an indicator that measures how good or bad a specific action is in a specific state relative to the average situation. The GAE (Generalized Advantage Estimation) method can be used to calculate the advantage function.

[0081] The strategy loss is calculated according to the advantage function, and the gradient descent algorithm is used to optimize the strategy loss. The model parameters of the initial decision model are updated until the preset stopping conditions are met. The decision model that meets the preset stopping conditions is used as the target decision model.

[0082] Among them, the strategy loss is used to measure the difference between the new strategy and the old strategy. The new strategy can be understood as the behavior decision at the next moment, and the old strategy can be understood as the behavior decision at the current moment. The preset stop condition may include the number of iterations reaching the preset number of iterations or the strategy loss meeting the preset strategy loss value.

[0083] In an optional embodiment, the method may further include:

[0084] S204: Train the initial value model according to the actual reward feedback and the predicted reward feedback to obtain the target value model.

[0085] According to the actual reward feedback and the predicted reward feedback at the next moment, the value loss is calculated. According to the value loss, the gradient descent algorithm is used to optimize the value loss and update the model parameters of the initial value model until the preset stopping conditions are met. The value model that meets the preset stopping conditions is used as the target value model.

[0086] Among them, the preset stop condition may include that the number of iterations reaches a preset number of iterations or the value loss satisfies a preset value loss value, and the value loss is used to measure the difference between the predicted reward feedback and the actual reward feedback of the value network.

[0087] Among them, the value loss (Value Loss) L V It can be expressed as:

[0088]

[0089] t is the next moment, s is the state, V(s t ) is the predicted reward feedback of the agent at time t, is the actual reward feedback of the agent at time t.

[0090] Among them, the advantage function A GAE (t) can be expressed as:

[0091]

[0092] δ t =R t +γV(s t+l )-V(s t ), l ranges from 0 to ∞, γ is the discount factor of reinforcement learning, ranging from 0.9 to 0.999, and λ is a parameter used to control the bias-variance trade-off in the Generalized Advantage Estimation (GAE). γ and λ can be selected according to actual conditions.

[0093] Policy Loss L CLIP It can be expressed as:

[0094] L CLIP =E[min(r(θ)A clip ,clip(r(θ),1-∈,1+∈)A]

[0095] Among them, E is the expected operation, r(θ) is the probability ratio of the new and old strategies, and A clip is the clipped advantage function, ∈ is the clipping parameter, and A is the advantage function.

[0096] It is worth noting that the output of the decision model is the probability value of the behavioral decision, and the ratio of the maximum probability value of the behavioral decision output by the decision model at time t to the maximum probability value of the behavioral decision at time t-1 is taken as the probability ratio of the new and old strategies.

[0097] In this embodiment, the decision model and value model in the reinforcement learning framework are combined for model training, which can automatically generate richer and more realistic intelligent agent behaviors.

[0098] Figure 3 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 3 ,like Figure 3 As shown, in an optional embodiment, the initial decision model includes: an initial structured model and a second preset large language model. The above step S103 processes the current state information according to the initial decision model of the agent to predict the next behavior decision of the agent in the game scene at the next moment, including:

[0099] S301. Perform structural processing on current state information according to an initial structural model to obtain structural information corresponding to the current state information.

[0100] The initial structured model is a trainable model, such as the LLAMA3 model, which is used to convert the original state information into vector information represented by numbers. The LLAMA3 model is based on a general pre-trained model with relatively few model parameters, and can be further customized and trained at low cost for specific tasks or scenarios.

[0101] The current state information is input into the initial structured model, keywords are extracted and structured, and the structured information corresponding to the current state information is obtained. For example, if the current state information is eating and chatting at the current time and place 1, the corresponding structured information can be current time: place 1: eating: chatting.

[0102] S302: Process the structured information according to the second preset large language model to predict the next behavior decision.

[0103] The second preset large language model may be the first preset large language model, and the structured information and the behavior decision prediction prompt information are input into the second preset large language model to predict the next behavior decision.

[0104] The second preset large language model is used to convert structured information into agent behavior decisions. The pre-trained large model is generally not retrained on its basis due to its huge parameters.

[0105] The above step S105, training the initial decision model according to the actual reward feedback to obtain the target decision model of the agent, may include:

[0106] S303: Train the initial structured model according to the actual reward feedback to obtain the target structured model.

[0107] Among them, the initial structured model is a trainable model, and the trainable base model parameters are usually in the billions to tens of billions, which is convenient for retraining into a model dedicated to group intelligent behavior decision-making. Therefore, according to the actual reward feedback, the initial structured model can be trained to obtain the target structured model. The target decision model includes: the target structured model and the second preset large language model.

[0108] In this embodiment, by performing structured processing on state information, knowledge and state descriptions that can guide decision-making are generated, providing necessary context and support for the decision-making process of the intelligent agent.

[0109] Figure 4 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 4 ,like Figure 4As shown, in an optional implementation, the above step S103, processing the current state information according to the initial decision model of the agent, and predicting the next behavior decision of the agent at the next moment, may include:

[0110] S401. According to the current state information, obtain the target historical state information of the agent from the preset memory library.

[0111] Among them, the preset memory library includes multiple historical state information of the intelligent agent at multiple historical moments, and the similarities between the current state information and the multiple historical state information in the preset memory library are calculated respectively, and the historical state information whose similarity exceeds the preset similarity threshold is used as the target historical state information.

[0112] S402: Process the current state information and the target historical state information according to the initial decision model to predict the next behavior decision.

[0113] The current state information and the target historical state information are spliced ​​to obtain a spliced ​​sample, which is input into the initial decision model to predict the next behavior decision.

[0114] In an optional implementation, the spliced ​​samples may be structured according to the initial structured model to obtain structured information corresponding to the spliced ​​samples, and the structured information may be processed according to the second preset large language model to predict the next behavior decision.

[0115] In this embodiment, the preset memory library stores the historical memory information of the intelligent agent, which can help the intelligent agent make more intelligent decisions. By determining the target historical state information from the preset memory library, the amount of input experience data (historical state information) can be reduced, thereby obtaining the memory related to the current state information, and predicting the next behavior decision based on the current state information and the memory related to the current state information, thereby improving the prediction accuracy of the behavior decision.

[0116] Figure 5 Schematic diagram of the process of the intelligent agent model training method provided in the embodiment of the present application Figure 5 ,like Figure 5 As shown, in an optional implementation, the above step S201, processing the interaction state according to the initial value model of the agent to obtain the predicted reward feedback of the agent at the next moment, may include:

[0117] S501. According to the interaction state, obtain the target historical interaction state of the agent from the preset memory library.

[0118] Among them, the preset memory library includes multiple historical state information of the intelligent agent at multiple historical moments, and the historical state information includes the interaction state between the intelligent agent and other intelligent agents. The similarities between the interaction state and the multiple historical interaction states in the preset memory library are calculated respectively, and the historical interaction state whose similarity exceeds the preset similarity threshold is taken as the target historical interaction state.

[0119] S502: Process the interaction state and the target historical interaction state according to the initial value model to obtain predicted reward feedback.

[0120] The interaction state and the target historical interaction state are spliced ​​to obtain a spliced ​​sample, and the spliced ​​sample is input into the initial value model to obtain the predicted reward feedback.

[0121] In this embodiment, by determining the target historical interaction state from a preset memory library, the amount of input experience data (historical state information) can be reduced, thereby obtaining memories related to the interaction state, and predicting reward feedback based on the interaction state and the memories related to the interaction state, thereby improving the prediction accuracy of the reward feedback.

[0122] Figure 6 A flow chart of the intelligent agent control method provided in the embodiment of the present application is shown as follows: Figure 6 As shown, the method may include:

[0123] S601, obtaining the state information of the agent in the game scene at the target time.

[0124] When controlling an agent to perform a behavior in a game scene, first obtain the state information of the agent in the game scene at a target time, where the target time is the previous time of the time to be controlled. The state information may include the position, environment, and behavior of the agent in the game scene at the target time, where the behavior may be, for example, sleeping, cooking, chatting, or exercising.

[0125] S602: Process the state information at the target moment according to the target decision model of the intelligent agent to obtain the next behavior decision of the intelligent agent at the next moment after the target moment.

[0126] S603: Based on the next behavior decision, control the agent to perform the corresponding behavior action at the next moment of the target moment.

[0127] The state information at the target moment is input into the target decision model of the intelligent agent to obtain the next behavior decision of the intelligent agent at the next moment after the target moment, and according to the next behavior decision, the intelligent agent is controlled to perform the corresponding behavior action at the next moment after the target moment to control the behavior action of the intelligent agent in the game scene.

[0128] In this embodiment, a more abundant and realistic agent behavior is automatically generated in combination with a target decision model.

[0129] Based on the above embodiments, the solution of the present application is described below in conjunction with a specific embodiment.

[0130] Figure 7 A schematic diagram of the agent model training process provided in the embodiment of the present application, such as Figure 7 As shown, it includes the following components:

[0131] Pre-trained large language models (such as GPT4): provide basic cognitive capabilities for other systems; model parameters are fixed.

[0132] Game swarm intelligence environment: A game environment where agents can interact and make decisions.

[0133] Trainable models (such as LLAMA3): decision models based on general pre-trained models with relatively few model parameters, which can be further customized and trained at low cost for specific tasks or scenarios.

[0134] Memory: Stores the agent's historical memory information to help the agent make more informed decisions; historical memory information. By combining the environment state output and general pre-training model processing, the most important memory set for future decision-making is obtained.

[0135] Agent behavior decision: The agent decides the next action based on the information and model it receives.

[0136] Knowledge and state descriptions that guide decision-making: This information provides the necessary context and support for the agent's decision-making process.

[0137] Retrieve the memory that is most helpful for the current decision: The agent is able to extract relevant information from the memory bank to assist its current decision making.

[0138] In this embodiment, the samples consisting of collection status, behavior decision and feedback can be stored in the training sample collection area, and the relevant information and samples in the memory bank are combined into a complete sample for training. During model training, the network is initialized, and the advantage function, gradient optimization strategy function and gradient optimization strategy function are calculated in turn until the preset stop condition is met to obtain the target decision model and the target value model.

[0139] Figure 8 This is a schematic diagram of the structure of an intelligent agent model training device provided in an embodiment of the present application, which can be integrated into an electronic device.

[0140] like Figure 8 As shown, the device may include:

[0141] The acquisition module 701 is used to obtain the current state information of the agent in the game scene at the current moment;

[0142] Processing module 702, used to process the current state information according to the initial decision model of the agent, and predict the next behavior decision of the agent in the game scene at the next moment;

[0143] The acquisition module 701 is also used to obtain the next state information of the agent in the game scene when executing the next behavior decision at the next moment, and the next state information includes: the interaction state of the agent and other agents;

[0144] The acquisition module 701 is also used to obtain actual reward feedback when the agent performs the next behavior decision according to the interaction state;

[0145] The processing module 702 is also used to train the initial decision model according to the actual reward feedback to obtain the target decision model of the intelligent agent.

[0146] In an optional implementation, the acquisition module 701 is specifically configured to:

[0147] According to the interaction state and the reward feedback rule information based on the natural language description, the first preset large language model is adopted to obtain the actual reward feedback.

[0148] In an optional implementation, the processing module 702 is further configured to:

[0149] Process the interaction state according to the agent's initial value model to obtain the agent's predicted reward feedback at the next moment;

[0150] The processing module 702 is specifically configured to:

[0151] Calculate the advantage function based on the actual reward feedback and the predicted reward feedback;

[0152] The initial decision model is trained according to the advantage function to obtain the target decision model.

[0153] In an optional implementation, the processing module 702 is further configured to:

[0154] According to the actual reward feedback and the predicted reward feedback, the initial value model is trained to obtain the target value model.

[0155] In an optional implementation, the initial decision model includes: an initial structured model and a second preset large language model; a processing module 702, specifically configured to:

[0156] According to the initial structured model, the current state information is structured to obtain structured information corresponding to the current state information;

[0157] According to the second preset large language model, the structured information is processed to predict the next behavior decision;

[0158] According to actual reward feedback, the initial structured model is trained to obtain a target structured model. The target decision model includes: a target structured model and a second preset large language model.

[0159] In an optional implementation, the processing module 702 is specifically configured to:

[0160] According to the current state information, the target historical state information of the agent is obtained from the preset memory library;

[0161] The current state information and target historical state information are processed according to the initial decision model to predict the next behavior decision.

[0162] In an optional implementation, the processing module 702 is specifically configured to:

[0163] According to the interaction state, the target historical interaction state of the agent is obtained from the preset memory library;

[0164] The interaction state and the target historical interaction state are processed according to the initial value model to obtain the predicted reward feedback.

[0165] In this embodiment, the acquisition module is used to obtain the current state information of the agent in the game scene at the current moment, the processing module is used to process the current state information according to the initial decision model of the agent, and predict the next behavior decision of the agent in the game scene at the next moment. The acquisition module is also used to obtain the next state information of the agent in the game scene when it performs the next behavior decision at the next moment. The next state information includes: the interaction state of the agent and other agents. The acquisition module is also used to obtain the actual reward feedback when the agent performs the next behavior decision according to the interaction state. The processing module is also used to train the initial decision model according to the actual reward feedback to obtain the target decision model of the agent. The decision model is trained in combination with the reward feedback in reinforcement learning, and the decision model of the agent is automatically generated, thereby improving the authenticity and diversity of the agent behavior in the game group environment.

[0166] Fig. 9 This is a schematic diagram of the structure of an intelligent body control device provided in an embodiment of the present application, which can be integrated into an electronic device.

[0167] like Fig. 9 As shown, the device may include:

[0168] The acquisition module 801 is used to obtain the state information of the agent in the game scene at the target time;

[0169] Processing module 802, used to process the state information at the target time according to the agent control model to obtain the next behavior decision of the agent at the next moment after the target time;

[0170] The processing module 802 is used to control the agent to perform corresponding behavior actions in the game scene according to the next behavior decision.

[0171] In this embodiment, the acquisition module is used to obtain the state information of the agent at the target time in the game scene, and the processing module is used to process the state information at the target time according to the agent control model to obtain the next behavior decision of the agent at the next moment of the target time. The processing module is used to control the agent to perform the corresponding behavior action in the game scene according to the next behavior decision. Combined with the target decision model, more rich and realistic agent behaviors are automatically generated.

[0172] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the device may include: a processor 901, a memory 902 and a bus 903, the memory 902 stores machine-readable instructions executable by the processor 901, when the electronic device is running, the processor 901 communicates with the memory 902 through the bus 903, and the processor 901 executes the machine-readable instructions to perform the following steps:

[0173] Get the current state information of the agent in the game scene at the current moment;

[0174] Process the current state information according to the initial decision model of the intelligent agent and predict the next behavior decision of the intelligent agent at the next moment;

[0175] Obtain the next state information of the agent when it performs the next behavior decision at the next moment. The next state information includes: the interaction state of the agent and other agents;

[0176] According to the interaction state, obtain the actual reward feedback when the agent makes the next action decision;

[0177] According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the intelligent agent.

[0178] In an optional implementation, obtaining actual reward feedback when the agent performs the next behavior decision according to the interaction state includes:

[0179] According to the interaction state and the reward feedback rule information based on the natural language description, the first preset large language model is adopted to obtain the actual reward feedback.

[0180] In an optional embodiment, the method further comprises:

[0181] Process the interaction state according to the agent's initial value model to obtain the agent's predicted reward feedback at the next moment;

[0182] According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the agent, including:

[0183] Calculate the advantage function based on the actual reward feedback and the predicted reward feedback;

[0184] The initial decision model is trained according to the advantage function to obtain the target decision model.

[0185] In an optional embodiment, the method further comprises:

[0186] According to the actual reward feedback and the predicted reward feedback, the initial value model is trained to obtain the target value model.

[0187] In an optional implementation, the initial decision model includes: an initial structured model and a second preset large language model; processing the current state information according to the initial decision model of the agent, and predicting the next behavior decision of the agent in the game scene at the next moment, including:

[0188] According to the initial structured model, the current state information is structured to obtain structured information corresponding to the current state information;

[0189] According to the second preset large language model, the structured information is processed to predict the next behavior decision;

[0190] According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the agent, including:

[0191] According to actual reward feedback, the initial structured model is trained to obtain a target structured model. The target decision model includes: a target structured model and a second preset large language model.

[0192] In an optional implementation, the current state information is processed according to the initial decision model of the agent to predict the next behavior decision of the agent at the next moment, including:

[0193] According to the current state information, the target historical state information of the agent is obtained from the preset memory library;

[0194] The current state information and target historical state information are processed according to the initial decision model to predict the next behavior decision.

[0195] In an optional implementation, the interaction state is processed according to the initial value model of the agent to obtain the predicted reward feedback of the agent at the next moment, including:

[0196] According to the interaction state, the target historical interaction state of the agent is obtained from the preset memory library;

[0197] The interaction state and the target historical interaction state are processed according to the initial value model to obtain the predicted reward feedback.

[0198] Alternatively, perform the following steps:

[0199] Get the state information of the agent in the game scene at the target time;

[0200] The state information at the target moment is processed according to the target decision model of the intelligent agent to obtain the next behavior decision of the intelligent agent at the next moment after the target moment;

[0201] According to the next behavior decision, the intelligent agent is controlled to perform the corresponding behavior action at the next moment of the target moment.

[0202] In this embodiment, the processor executes machine-readable instructions to obtain the current state information of the agent in the game scene at the current moment, process the current state information according to the initial decision model of the agent, predict the next behavior decision of the agent in the game scene at the next moment, and obtain the next state information of the agent when executing the next behavior decision at the next moment. The next state information includes: the interaction state of the agent and other agents, and according to the interaction state, obtain the actual reward feedback when the agent executes the next behavior decision. According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the agent. The decision model is trained in combination with the reward feedback in reinforcement learning, and the decision model of the agent is automatically generated, thereby improving the authenticity and diversity of the agent behavior in the game group environment.

[0203] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps:

[0204] Get the current state information of the agent in the game scene at the current moment;

[0205] Process the current state information according to the initial decision model of the intelligent agent and predict the next behavior decision of the intelligent agent at the next moment;

[0206] Obtain the next state information of the agent when it performs the next behavior decision at the next moment. The next state information includes: the interaction state of the agent and other agents;

[0207] According to the interaction state, obtain the actual reward feedback when the agent makes the next action decision;

[0208] According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the intelligent agent.

[0209] In an optional implementation, obtaining actual reward feedback when the agent performs the next behavior decision according to the interaction state includes:

[0210] According to the interaction state and the reward feedback rule information based on the natural language description, the first preset large language model is adopted to obtain the actual reward feedback.

[0211] In an optional embodiment, the method further comprises:

[0212] Process the interaction state according to the agent's initial value model to obtain the agent's predicted reward feedback at the next moment;

[0213] According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the agent, including:

[0214] Calculate the advantage function based on the actual reward feedback and the predicted reward feedback;

[0215] The initial decision model is trained according to the advantage function to obtain the target decision model.

[0216] In an optional embodiment, the method further comprises:

[0217] According to the actual reward feedback and the predicted reward feedback, the initial value model is trained to obtain the target value model.

[0218] In an optional implementation, the initial decision model includes: an initial structured model and a second preset large language model; processing the current state information according to the initial decision model of the agent, and predicting the next behavior decision of the agent in the game scene at the next moment, including:

[0219] According to the initial structured model, the current state information is structured to obtain structured information corresponding to the current state information;

[0220] According to the second preset large language model, the structured information is processed to predict the next behavior decision;

[0221] According to the actual reward feedback, the initial decision model is trained to obtain the target decision model of the agent, including:

[0222] According to actual reward feedback, the initial structured model is trained to obtain a target structured model. The target decision model includes: a target structured model and a second preset large language model.

[0223] In an optional implementation, the current state information is processed according to the initial decision model of the agent to predict the next behavior decision of the agent at the next moment, including:

[0224] According to the current state information, the target historical state information of the agent is obtained from the preset memory library;

[0225] The current state information and target historical state information are processed according to the initial decision model to predict the next behavior decision.

[0226] In an optional implementation, the interaction state is processed according to the initial value model of the agent to obtain the predicted reward feedback of the agent at the next moment, including:

[0227] According to the interaction state, the target historical interaction state of the agent is obtained from the preset memory library;

[0228] The interaction state and the target historical interaction state are processed according to the initial value model to obtain the predicted reward feedback.

[0229] Alternatively, perform the following steps:

[0230] Get the state information of the agent in the game scene at the target time;

[0231] The state information at the target moment is processed according to the target decision model of the intelligent agent to obtain the next behavior decision of the intelligent agent at the next moment after the target moment;

[0232] According to the next behavior decision, the intelligent agent is controlled to perform the corresponding behavior action at the next moment of the target moment.

[0233] In this embodiment, when the computer program is executed by the processor, the processor executes to obtain the current state information of the agent in the game scene at the current moment, processes the current state information according to the initial decision model of the agent, predicts the next behavior decision of the agent in the game scene at the next moment, obtains the next state information when the agent performs the next behavior decision at the next moment, and the next state information includes: the interaction state of the agent and other agents, obtains the actual reward feedback when the agent performs the next behavior decision according to the interaction state, and trains the initial decision model according to the actual reward feedback to obtain the target decision model of the agent. The decision model is trained in combination with the reward feedback in reinforcement learning, and the decision model of the agent is automatically generated, thereby improving the authenticity and diversity of the agent behavior in the game group environment.

[0234] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0235] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0236] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0238] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0239] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0240] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for training an intelligent agent model, characterized in that: include: Get the current state information of the agent in the game scene at the current moment; Processing the current state information according to the initial decision model of the agent to predict the next behavior decision of the agent at the next moment; Acquire next state information of the agent when executing the next behavior decision at the next moment, the next state information including: interaction state of the agent and other agents; According to the interaction state, obtaining actual reward feedback when the agent executes the next behavior decision; The initial decision model is trained according to the actual reward feedback to obtain the target decision model of the agent.

2. The method according to claim 1, characterized in that The step of obtaining actual reward feedback when the agent executes the next behavior decision according to the interaction state includes: According to the interaction state and the reward feedback rule information based on the natural language description, a first preset large language model is adopted to obtain the actual reward feedback.

3. The method according to claim 1, characterized in that The method further comprises: Processing the interaction state according to the initial value model of the agent to obtain predicted reward feedback of the agent at the next moment; The step of training the initial decision model according to the actual reward feedback to obtain the target decision model of the agent includes: Calculating an advantage function according to the actual reward feedback and the predicted reward feedback; The initial decision model is trained according to the advantage function to obtain the target decision model.

4. The method according to claim 3, characterized in that The method further comprises: The initial value model is trained according to the actual reward feedback and the predicted reward feedback to obtain a target value model.

5. The method according to claim 1, characterized in that The initial decision model includes: an initial structured model and a second preset large language model; the current state information is processed according to the initial decision model of the intelligent agent to predict the next behavior decision of the intelligent agent at the next moment, including: According to the initial structured model, the current state information is structured to obtain structured information corresponding to the current state information; Processing the structured information according to the second preset large language model to predict the next action decision; The step of training the initial decision model according to the actual reward feedback to obtain the target decision model of the agent includes: The initial structured model is trained according to the actual reward feedback to obtain a target structured model, and the target decision model includes: the target structured model and the second preset large language model.

6. The method according to claim 1, characterized in that The processing of the current state information according to the initial decision model of the intelligent agent to predict the next behavior decision of the intelligent agent at the next moment includes: According to the current state information, obtaining the target historical state information of the agent from a preset memory library; The current state information and the target historical state information are processed according to the initial decision model to predict the next behavior decision.

7. The method according to claim 3, characterized in that The step of processing the interaction state according to the initial value model of the agent to obtain the predicted reward feedback of the agent at the next moment includes: According to the interaction state, obtaining the target historical interaction state of the agent from a preset memory library; The interaction state and the target historical interaction state are processed according to the initial value model to obtain the predicted reward feedback.

8. An intelligent agent control method, characterized in that: include: Get the state information of the agent in the game scene at the target time; Processing the state information at the target moment according to the target decision model of the agent to obtain the next behavior decision of the agent at the next moment after the target moment; According to the next behavior decision, the agent is controlled to perform a corresponding behavior action at the next moment after the target moment.

9. An intelligent agent control model training device, characterized in that: include: The acquisition module is used to obtain the current state information of the agent in the game scene at the current moment; A processing module, used to process the current state information according to the initial decision model of the intelligent agent, and predict the next behavior decision of the intelligent agent at the next moment; The acquisition module is further used to acquire the next state information of the agent when executing the next behavior decision at the next moment, and the next state information includes: the interaction state of the agent and other agents; The acquisition module is further used to acquire actual reward feedback when the agent executes the next behavior decision according to the interaction state; The processing module is also used to train the initial decision model according to the actual reward feedback to obtain the target decision model of the agent.

10. An intelligent agent control device, characterized in that: include: The acquisition module is used to obtain the state information of the agent in the game scene at the target time; A processing module, used for processing the state information of the target moment according to the intelligent agent control model to obtain the next behavior decision of the intelligent agent at the next moment after the target moment; The processing module is used to control the agent to perform corresponding behavioral actions in the game scene according to the next behavior decision.

11. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images