Method and apparatus for training artificial intelligence model, and device, medium and program product

Abstract

A method and apparatus for training an artificial intelligence (AI) model, and a device and a storage medium, which are used for quickly performing training to obtain an AI model and also improving the generalization of the AI model. The method comprises: acquiring an initial AI model and a first numerical interval, wherein the first numerical interval is a numerical value range of a plurality of pieces of attribute data corresponding to an opponent character of the initial AI model (201); calling the initial AI model to determine a second numerical interval from the first numerical interval (202); in the second numerical interval, performing random sampling on the plurality of pieces of attribute data to generate a training character set (203); and using the training character set to perform reinforcement learning training on the initial AI model, so as to obtain a target AI model (204).

Description

Training methods, devices, equipment, media and program products for artificial intelligence models

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 25, 2023, with application number 2023109238917 and application name “Training method, device, equipment and storage medium for artificial intelligence model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] This application relates to artificial intelligence, and in particular to AI model training. Background Art

[0003] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. Since its inception, AI theory and technology have matured, and its application areas have continued to expand. It is conceivable that future AI-enabled technological products will serve as "containers" of human wisdom. AI can simulate the information processes of human consciousness and thinking. While AI is not human intelligence, it can think like humans and may even surpass human intelligence.

[0004] Artificial intelligence can also be applied to gaming. For example, in open-world games, players can freely roam the virtual world and freely choose the events and methods to complete game tasks. Therefore, a large number of items and characters are needed to populate the game world. Specifically, the AI ​​models for non-player characters (NPCs) in open-world games are extremely important. Specifically, NPCs in open-world games require a wide range of decision-making spaces and a rich variety of strategies. During game interactions, NPCs must avoid risks through strategic movement and inflict maximum damage to opposing characters through targeted attacks. Given the diverse and varied strategies of opponents, formulating, selecting, and executing strategies is crucial for game intelligence systems, given the vast decision-making space and the real-time demands of decision-making.

[0005] Therefore, there are higher requirements for the training of game AI models for NPCs in the game.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a training method, apparatus, device, storage medium, and program product for an artificial intelligence model, which are used to quickly train an AI model while improving the generalization of the AI ​​model.

[0008] In view of this, on the one hand, the present application provides a training method for an artificial intelligence model, including: obtaining an initial artificial intelligence AI model and a first numerical interval, the first numerical interval being the numerical value range of multiple attribute data corresponding to the opponent character of the initial AI model; calling the initial AI model to determine a second numerical interval from the first numerical interval; randomly sampling the multiple attribute data in the second numerical interval to generate a training character set; and using the training character set to perform reinforcement learning training on the initial AI model to obtain a target AI model.

[0009] Another aspect of the present application provides a model training device, comprising: an acquisition module for acquiring an initial artificial intelligence (AI) model and a first numerical interval, the first numerical interval being a numerical value range of multiple attribute data corresponding to an opponent character of the initial AI model;

[0010] A processing module, configured to call the initial AI model to determine a second numerical interval from the first numerical interval; and randomly sample the plurality of attribute data in the second numerical interval to generate a training character set;

[0011] The training module is used to perform reinforcement learning training on the initial AI model using the training role set to obtain the target AI model.

[0012] Another aspect of the present application provides a computer device, comprising: a memory, a processor, and a bus system;

[0013] Wherein, the memory is used to store computer programs;

[0014] The processor is configured to execute a computer program in the memory, and the processor is configured to perform the above-mentioned methods according to the computer program;

[0015] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0016] Another aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0017] Another aspect of the present application provides a computer program product comprising a computer program, which, when executed on a computer, enables the computer to execute the above method.

[0018] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages: using reinforcement learning to train the AI ​​model can reduce the manual maintenance of the model and achieve rapid model training. At the same time, during the training process, the attribute data of the combat characters are numerically limited to ensure the validity of the sample data during the AI ​​model training process. At the same time, random sampling is performed within the selected numerical range to generate combat characters, making the sample data more extensive, which can improve the generalization of the AI ​​model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a schematic diagram of an architecture of a training system according to an embodiment of the present application;

[0020] FIG2 is a schematic diagram of an embodiment of a method for training an artificial intelligence model in an embodiment of the present application;

[0021] FIG2a is a schematic diagram of an iterative screening process of a numerical interval in an embodiment of the present application;

[0022] FIG2 b is a schematic diagram of an embodiment of the NPC control method in the embodiment of the present application;

[0023] FIG2c is a schematic diagram of an architecture of an NPC control system according to an embodiment of the present application;

[0024] FIG3 is a schematic diagram of an embodiment of an artificial intelligence model training device according to an embodiment of the present application;

[0025] FIG4 is a schematic diagram of another embodiment of the artificial intelligence model training device according to the embodiment of the present application;

[0026] Figure 5 is a schematic diagram of another embodiment of the artificial intelligence model training device in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application provide a training method, apparatus, device, and storage medium for an artificial intelligence model, which are used to quickly train an AI model while improving the generalization of the AI ​​model.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Since the quality of the NPC's game AI model directly affects the gaming experience, training has higher requirements.

[0030] In order to solve the above problems, this application provides a training method for an artificial intelligence model, so that the sample data can be more extensive, which can improve the generalization of the AI ​​model.

[0031] For ease of understanding, some nouns in this application are explained below.

[0032] Game AI (Artificial Intelligence, AI): Game AI is a program or character introduced into the game that combines artificial intelligence related technologies to enrich the gameplay and enhance the player's gaming experience.

[0033] Reinforcement Learning: Reinforcement learning is a type of machine learning in which a system learns from its environment to maximize rewards. Within the machine learning landscape, reinforcement learning focuses more on the interaction between an agent and its environment. Reinforcement learning consists of two main components: the agent and the environment. It also includes three elements: state or observation, action, and reward (also known as a reward function). The environment is an external system within which the agent resides, capable of perceiving it and taking actions based on its perceived state. The agent is a system embedded within the environment, capable of changing its state by taking actions. State / Observation: The state is a complete description of the world, concealing no information about the world. Observations are partial descriptions of the state, potentially omitting some information. Actions: Different environments allow different types of actions. The set of valid actions in a given environment is often called the action space, which includes discrete action spaces and continuous action spaces. For example, if a maze-navigating robot has only four possible movement options (east, south, west, and north), it has a discrete action space; if the robot can move in any direction within 360°, it has a continuous action space. Reward: A scalar feedback signal provided by the environment indicates how well the agent performed at a particular step using a particular strategy. Based on the above architecture, reinforcement learning has the following key characteristics: trial-and-error learning, meaning that reinforcement learning generally lacks direct guidance information. The agent must continuously interact with the environment to obtain the optimal policy through trial and error. Delayed rewards, meaning that guidance information in reinforcement learning is minimal and often provided only after the fact (at the last state). That is, the initial reinforcement learning model learns through continuous trial and error and feedback, and is often used to make sequential decisions or control problems, such as game AI and drones.

[0034] Open World Games: A type of video game where players can freely roam a virtual world and choose the events and methods to complete game tasks. This high degree of freedom often requires a large number of items and characters to populate the game world.

[0035] NPC (Non-Person Characters in Games): Non-human player characters in the game are an important part of many games.

[0036] Deep Learning: A neural network algorithm that uses multiple complex structures or nonlinear transformation processing layers to achieve better high-level abstraction capabilities than shallow neural networks.

[0037] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental path to computer intelligence. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. With the advancement of AI technology, research and application are expanding across a wide range of fields, including smart homes, smart wearables, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robotics, smart healthcare, and smart customer service. With the advancement of technology, AI will be applied in even more fields and play an increasingly important role.

[0038] The embodiments of the present application provide a method, apparatus, device, and storage medium for training an artificial intelligence model, which can quickly train an AI model while improving the generalization of the AI ​​model. The following describes exemplary applications of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals or as a server.

[0039] By running the artificial intelligence model training method provided in the embodiments of the present application, electronic devices can quickly train AI models and improve the generalization of AI models, that is, improve the electronic device's ability to quickly and efficiently update AI models. This can be applied to multiple application scenarios such as gaming scenarios and AI intelligent customer service. For example, fighting games, multiplayer online battle arena (MOBA), or shooting games in gaming scenarios.

[0040] Referring to Figure 1, Figure 1 is an optional architectural diagram of an application scenario of the training method of the artificial intelligence model provided in an embodiment of the present application. To implement the training method supporting an artificial intelligence model, the terminal device 100 is connected to the server 300 via the network 200, and the server 300 is connected to the database 400. The network 200 can be a wide area network or a local area network, or a combination of the two. The game client is deployed on the terminal device 100, wherein the client can be run on the terminal device 100 in the form of a browser, or can be run on the terminal device 100 in the form of an independent application (application, APP), etc. The specific presentation form of the client is not limited here. The server 300 involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 100 may be a smartphone, tablet computer, laptop computer, PDA, personal computer, smart TV, smart watch, in-vehicle device, wearable device, etc., but is not limited thereto. The terminal device 100 and the server 300 may be connected directly or indirectly via the network 200 via wired or wireless communication, which is not limited in this application. The number of servers 300 and terminal devices 100 is also not limited. The solution provided in this application can be implemented independently by the terminal device 100, independently by the server 300, or in cooperation with the terminal device 100 and the server 300, which is not specifically limited in this application. In short, the database 400 can be considered an electronic filing cabinet—a place where electronic files are stored, where users can add, query, update, delete, and perform other operations on the data in the files. The so-called "database" is a collection of data stored in a certain manner, which can be shared by multiple users, has minimal redundancy, and is independent of the application program. A database management system (DBMS) is a computer software system designed for managing databases, generally providing basic functions such as storage, interception, security, and backup.Database management systems can be classified based on the database model they support, such as relational or Extensible Markup Language (XML); or based on the type of computers they support, such as server clusters and mobile phones; or based on the query language used, such as Structured Query Language (SQL) or XQuery; or based on performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​at the same time. In the present application, database 400 can be used to store the interaction data of two characters in the current interaction situation. Of course, the storage location of the interaction data of the two interacting characters in the current interaction situation is not limited to the database. For example, it can also be stored in the terminal device 100, the blockchain, or the distributed file system of the server 300.

[0041] In some embodiments, the server 300 can jointly execute the training method of the artificial intelligence model provided by the embodiment of the present application with the terminal device 100. In this embodiment, the specific process can be as follows: the terminal device 100 obtains an initial artificial intelligence AI model and a first numerical interval, where the first numerical interval is a numerical value range of multiple attribute data corresponding to the opponent character of the initial AI model; calls the initial AI model to determine a second numerical interval from the first numerical interval; randomly samples the multiple attribute data in the second numerical interval to generate a training character set; then the terminal device 100 calls the initial AI model to control the non-player character to interact with the character of the training character set to obtain sample data, and then the sample data is collected. The data can be stored in the database 400 or in the memory of the terminal device 100; the server 300 trains the initial AI model from the sample data stored in the database 400 or the terminal device 100 to obtain the target AI model; finally, the server 300 can deploy the target AI model to the terminal device 100, so that the terminal device 100 can call the target AI model to control the non-player character to achieve combat or other operations in the interactive scenario; or, the server 300 deploys the target AI model on the server corresponding to the client, so that the terminal device 100 can call the target AI model to the client's server to control the non-player character to achieve combat or other operations in the interactive scenario.

[0042] In another embodiment, the terminal device 100 independently executes the information recommendation method provided by the embodiment of the present application. In this embodiment, the specific process can be as follows: the terminal device 100 obtains an initial artificial intelligence AI model and a first numerical interval, where the first numerical interval is a numerical value range of multiple attribute data corresponding to the opponent character of the initial AI model; calls the initial AI model to determine a second numerical interval from the first numerical interval; randomly samples the multiple attribute data in the second numerical interval to generate a training character set; then the terminal device 100 calls the initial AI model to control the non-player character to interact with the character in the training character set to obtain sample data, and then the sample data can be stored in the database 400 or in the memory of the terminal device 100; the terminal device 100 again trains the initial AI model from the database 400 or the sample data stored in the terminal device 100 to obtain the target AI model. Finally, the terminal device 100 can deploy the target AI model to the terminal device 100, so that the terminal device 100 can call the target AI model to control the battle or other operations in the non-player character implementation; or, the terminal device 100 deploys the target AI model on the server corresponding to the client, so that the terminal device 100 can call the target AI model to the server of the client to control the battle or other operations in the non-player character implementation.

[0043] It is understandable that in the specific implementation of this application, when it comes to attribute data, interaction data and other related data, when the above embodiments of this application are applied to specific products or technologies, any one of them needs to obtain user permission or consent separately, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0044] In combination with the above introduction, the training method of the artificial intelligence model in this application is introduced below with the terminal device as the execution subject. Please refer to Figure 2. An embodiment of the training method of the artificial intelligence model in the embodiment of this application includes:

[0045] 201. Obtain an initial artificial intelligence (AI) model and a first numerical range.

[0046] The initial AI model is used to control the target character, and the first numerical interval is the numerical value range of multiple attribute data corresponding to the opponent character, and the opponent character is used to interact with the target character.

[0047] The target character and the opponent character are characters in the same virtual scene. They can interact with each other in the virtual scene, such as by asking and answering questions in a virtual question-and-answer scene or playing games in a virtual game scene, etc., which is not limited in this application. The target character's interactive behavior is controlled by the initial AI model or the subsequent target AI model. Its interactive behavior can be passive in response to the opponent character's interactive behavior or can be actively directed to the opponent character.

[0048] For ease of explanation, in the following embodiments, a non-player character is mainly used as an example of the target character for explanation.

[0049] In this embodiment, before training begins, the terminal device constructs an initial AI model for controlling the non-player character to perform corresponding interactive operations and obtains an initial numerical range (i.e., the first numerical range) for the attribute data of each character in the set of characters interacting with the non-player character. This attribute data can be understood as qualitative or quantitative variables of the non-player character in multiple dimensions. For example, using a game scenario as an example, before training begins, the terminal device constructs an initial game AI model for controlling the non-player character to perform corresponding game operations and obtains an initial numerical range for the attribute data of each character in the set of game characters interacting with the non-player character. It is understood that in this application scenario, this attribute data can be information such as the game character's attack power, defense power, and health pool. For example, the numerical range for attack power is set to (0, 100), the numerical range for defense power is set to (0, 100), and the numerical range for health pool is set to (0, 100).

[0050] In this embodiment, the initial AI model can be a deep neural network structure. In the deep neural network structure, it generally includes an input layer, a hidden layer, and an output layer. Among them, the input layer is the first layer of the deep neural network, the output layer is the last layer of the deep neural network, and the intermediate layers are all hidden layers, wherein multiple hidden layers can increase the expressive power of the model. In the deep neural network, each layer is fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. In this embodiment, the information input by the input layer is the sample data corresponding to each game character in the game character set, and the hidden layer is used to extract features from the sample data and express features of the sample data; then, after passing through multiple hidden layers, the predicted action probability distribution corresponding to the sample data is obtained, and the predicted action is obtained based on the predicted probability distribution.

[0051] 202. Call the initial AI model to determine a second numerical interval from the first numerical interval.

[0052] Among them, the second numerical range is adapted to the target role controlled by the initial AI model.

[0053] Since it is necessary to subsequently generate a training character set through a second numerical interval to perform reinforcement learning training on the initial AI model, and the target object may not be adapted to the interaction of all characters (first numerical interval) in the virtual scene, or will not have the opportunity to interact with characters with specific numerical values ​​in the virtual scene, in order to improve the training quality of the initial AI model, it is necessary to specifically select a second data interval that is adapted to the target character from the first numerical interval for the target character controlled by the initial AI model.

[0054] In this embodiment, the terminal device may perform the following operations when calling the initial AI model to determine the second numerical interval from the first numerical interval:

[0055] Randomly sampling the multiple attribute data in the first numerical interval to generate a first character set; using the first character set to perform reinforcement learning training on the initial AI model to obtain a first AI model; randomly sampling the multiple attribute data in the first numerical interval to generate a second character set; calling the first AI model to control the non-player characters to interact with the second character set to generate a first evaluation result; calculating a first screening condition value of the first AI model based on the first evaluation result; determining a third numerical interval of the multiple attribute data based on the first screening condition value and a preset screening threshold; randomly sampling the multiple attribute data in the third numerical interval to generate a third character set; using the third character set to perform reinforcement learning training on the initial AI model to obtain a second AI model; randomly sampling the multiple attribute data in the third numerical interval to generate a fourth character set; calling the second AI model to control the non-player characters to interact with the fourth character set to generate a second evaluation result; calculating a second screening condition value of the second AI model based on the second evaluation result; determining a fourth numerical interval of the multiple attribute data based on the second screening condition value and the preset screening threshold; repeating the above operations until the numerical intervals of the multiple attribute data reach a convergence condition, and outputting the converged numerical interval as the second numerical interval.

[0056] In an exemplary solution, the above process can be explained by taking a game scene as an example, as shown in FIG2a:

[0057] First, the terminal device randomly samples multiple attribute data within a first numerical range to generate a first set of game characters. Each game character in the first set of game characters is used as a training set for training the initial game AI model. The initial game AI model is then called to control an NPC to interact with each game character in the first set of game characters to generate sample data for this training process. The initial game AI model is then trained based on the sample data to obtain a first game AI model. Multiple attribute data within the first numerical range are again randomly sampled to generate a second set of game characters, which serves as an evaluation test set. The first game AI model is then called to control the NPC to interact with each game character in the second set of game characters again to generate a first evaluation result. A first screening condition value for the first game AI model is then calculated based on the first evaluation result. Finally, a third numerical range for the multiple attribute data is determined based on the first screening condition value and a preset screening threshold. A second round of iteration is then performed, with the third numerical range serving as the initial numerical range. The above steps are repeated until the numerical range reaches a convergence condition, and the final numerical range output is used as the second numerical range.

[0058] That is to say, in order to accurately filter out the second numerical range suitable for the target role from the first numerical range, the embodiment of the present application generates a role set within a numerical range, and allows the initial AI model to control the target role to actually interact with it, and uses the generated evaluation results as the basis for screening the numerical range. Through multiple rounds of iterations, the numerical range is gradually adjusted to obtain an accurate second numerical range suitable for the target role.

[0059] In this application, the random sampling can be understood as random value selection. For example, assuming that multiple attribute data are set to attack power, defense power and health volume, and the first numerical interval is set to: the numerical interval of attack power is (0, 100), the numerical interval of defense power is (0, 100) and the numerical interval of health volume is (0, 100). Therefore, the random sampling process can be understood as randomly selecting values ​​from the numerical intervals of the above three attribute data each time to form the game character. For example, for the first time, a value of 80 is selected from the numerical interval of the attack power, a value of 85 is selected from the numerical interval of the defense power, and a value of 90 is selected from the numerical interval of the health volume. Then, the game character formed at this time is game character A (attack power: 80, defense power: 85, health volume: 90).

[0060] Optionally, when calling the first AI model to control the non-player character to interact with the second character set to generate a first evaluation result, the following technical solution can be adopted: calling the first AI model to control the non-player character to interact with N characters in the second character set M times to obtain N battle data sets, wherein each battle data set includes M groups of battle data, and the battle data includes but is not limited to battle results and battle duration. Each character in the second character set corresponds to a set of attribute data, and M and N are positive integers; the N battle data sets are combined as the first evaluation result.

[0061] In an exemplary scheme, a game scenario is used as an example. That is, when the terminal device calls the first game AI model to control the NPC to interact with the second game character set to generate the first evaluation result, the following technical scheme can be adopted: call the first game AI model to control the non-player game character to interact with the N game characters in the second game character set M times to obtain N battle data sets, wherein each battle data set includes M groups of battle data, and the battle data includes but is not limited to battle results and battle duration. Each game character in the second game character set corresponds to a set of attribute data, and M and N are positive integers; the N battle data sets are combined as the first evaluation result.

[0062] For example, when the terminal device randomly samples the multiple attribute data in the first numerical interval to generate the second game character set, assuming that the multiple attribute data and the numerical interval can be set as follows: attack power and its corresponding first numerical interval is (0, 100), defense power and its corresponding first numerical interval is (0, 100), and health and its corresponding first numerical interval is (0, 100); then the second game character set generated by randomly sampling in the first numerical interval may include: game character 1 (attack power: 10, defense power: 80, health: 100), game character 2 (attack power: 1, defense power: 98, health: 85), game character 3 (attack power: 55, defense power: 89, health: 90), game character 4 (attack power: 10, defense power: 12, health: 17), game character 5 (attack power: 99, defense power: 18, health: 90). Then, the first game AI model is called to control the NPC to interact with game characters 1 to 5 10 times respectively to generate 5 battle data sets. For example, the battle data set generated by the first game AI model controlling the NPC to interact with the game character 1 10 times can be as follows: (battle result: NPC victory, battle time: 20 seconds), (battle result: NPC victory, battle time: 20 seconds), (battle result: NPC victory, battle time: 30 seconds), (battle result: NPC victory, battle time: 15 seconds), (battle result: NPC victory, battle time: 10 seconds) (battle result: NPC victory, battle time: 25 seconds), (battle result: NPC failure, battle time: 35 seconds), (battle result: NPC failure, battle time: 40 seconds), (battle result: NPC victory, battle time: 35 seconds), (battle result: NPC failure, battle time: 20 seconds). Similarly, the form of the battle data set generated by the first game AI model controlling the NPC to interact with the game characters 2 to 5 10 times can be as above, and the details will not be repeated here.

[0063] Since the interaction results of isolated cases are highly random, in each round of iteration, the impact of isolated cases is minimized by increasing the number of opponent characters used to train the initial AI model and the number of interaction games. This makes the evaluation results closer to the actual interactions in most cases, thereby improving the accuracy of determining the adaptability of the target character.

[0064] Optionally, after the terminal device obtains the first evaluation result, the method for the terminal device to calculate the first screening condition value of the first AI model based on the first evaluation result can be as follows: statistical processing is performed on each of the N battle data sets in turn to obtain N groups of battle result parameters of the first AI model for the second character set, wherein the N groups of battle result parameters serve as the first screening condition value. The battle result parameters may include various information describing the battle results, such as the battle result (win, lose, draw), battle duration, number of skill releases during the battle, damage value, etc., which are not limited in this application.

[0065] In an exemplary scheme, taking a game scenario as an example, after the terminal device obtains the first evaluation result, the method for the terminal device to calculate the first filtering condition value of the first game AI model based on the first evaluation result can be as follows: statistical processing is performed on each of the N battle data sets in turn to obtain the N groups of match win rates and battle durations of the first game AI model for the second character set, wherein the N groups of match win rates and battle durations serve as the first filtering condition values. In this application scenario, assuming that the first game AI model controls the NPC to interact with the game character 1 10 times, the battle data set generated can be as follows: (battle result: NPC victory, battle time: 20 seconds), (battle result: NPC victory, battle time: 20 seconds), (battle result: NPC victory, battle time: 30 seconds), (battle result: NPC victory, battle time: 15 seconds), (battle result: NPC victory, battle time: 10 seconds) (battle result: NPC victory, battle time: 25 seconds), (battle result: NPC failure, battle time: 35 seconds), (battle result: NPC failure, battle time: 40 seconds), (battle result: NPC victory, battle time: 35 seconds), (battle result: NPC failure, battle time: 20 seconds). The filtering condition value for game character 1 can then be: the battle win rate is 7 / 10 = 0.7, and the battle duration is (20+20+30+15+10+25+35+40+35+20) / 10 = 25 seconds. Similarly, the filtering condition values ​​for other game characters can be selected using the same algorithm, and the details will not be repeated here. It is understood that all the filtering condition values ​​for game characters 1 to 5 mentioned above will be used as the first filtering condition value.

[0066] The battle result parameters can intuitively reflect the interaction between the target character and the opponent character during the battle process. Based on the first evaluation result represented by the battle result parameters, through statistical processing, the adaptability of each battle data set and the target character can be reflected as a whole, thereby improving the accuracy and intuitiveness of the first screening condition value.

[0067] In one possible implementation, the battle result parameters include a battle win rate and a battle duration. The aforementioned statistical processing is performed on each of the N battle data sets in sequence to obtain N sets of battle result parameters of the first AI model for the second character set, including:

[0068] Performing probability statistics on battle results in a first battle data set to obtain the battle winning rate, where the first battle data set is included in the N battle data sets;

[0069] Averaging the battle times in the first battle data set to obtain the battle duration;

[0070] In this way, the N battle data sets are traversed to obtain N sets of victory rates and battle durations of the first AI model against the second character set.

[0071] In this embodiment, the battle result parameters are mainly characterized by the battle win rate and the battle duration, wherein the battle win rate can intuitively reflect the final result of the battle, and the battle duration can intuitively reflect the intensity of the battle. For example, a shorter battle may be a one-sided battle with a large difference in strength, and a longer battle may be a evenly matched battle with a smaller difference in strength.

[0072] By performing probability statistics on the battle results based on the winning rate and determining the average battle duration of the battle data set, and the winning rate and the average battle duration are important expression elements in the battle result parameters and are clearly quantified, the first battle data set can be represented in a concise and holistic manner through a small amount of data information, thereby improving the expression quality and accuracy of the first screening condition value.

[0073] Optionally, after obtaining the filtering condition value, the terminal device may determine the third numerical range from the first numerical range based on the filtering condition value and a preset filtering threshold. The specific operation may be as follows: traverse the N groups of battle result parameters, and determine X groups of battle result parameters from the N groups of battle result parameters based on the preset filtering threshold, where X is a positive integer; determine a fifth character set corresponding to the X groups of battle result parameters, where the fifth character set is included in the second character set; and determine the third numerical range based on multiple attribute data corresponding to each character in the fifth character set.

[0074] The preset filtering threshold is related to the data dimension involved in the first filtering condition value, that is, it is related to the data dimension involved in the battle result parameter. For example, when the first filtering condition value includes the battle win rate and the battle duration, the preset filtering threshold includes at least one of the specified battle win rate or the specified battle duration.

[0075] The preset screening threshold is used to screen out X roles that are more suitable for the target role from N roles according to the screening requirements. The specific threshold setting value may be related to the screening requirements or the screening accuracy.

[0076] By presetting the screening threshold, you can intuitively and quickly screen out opponent characters that are compatible with the target character in each round of iteration, thereby improving the screening accuracy.

[0077] In a possible implementation, the third value interval may be determined as follows:

[0078] Determine the maximum value and the minimum value of each attribute data corresponding to each character in the fifth character set; and determine the third numerical range according to the maximum value and the minimum value. For example, assuming that the character is a game character, and setting game character 1 (attack power: 10, defense power: 80, health: 100), game character 2 (attack power: 1, defense power: 98, health: 85), game character 3 (attack power: 55, defense power: 89, health: 90), game character 4 (attack power: 10, defense power: 12, health: 17), game character 5 (attack power: 99, defense power: 18, health: 90), and the calculated filtering condition values ​​​​of game character 1 to game character 5 are as follows: Game character 1: the winning rate is 0.7, and the battle time is 25 seconds; Game character 2: the winning rate is 0.20, and the battle time is 60 seconds; Game character 3: the winning rate is 0.6, and the battle time is 12 seconds; Game character 4: the winning rate is 0.9, and the battle time is 10 seconds; Game character 5: the winning rate is 0.7, and the battle time is 45 seconds. At the same time, the preset screening threshold is set to a win rate of 0.4 to 0.8, and the battle duration is 25 to 45. Based on the above screening condition values ​​and the preset screening threshold, the game characters screened out are: game character 1 and game character 5. At this time, based on the attribute data corresponding to game character 1 and game character 5, the third value range can be determined as follows: attack power (10, 99); defense power (18, 80); health (90, 100).

[0079] Since the characters in the fifth character set are all opponent characters that are compatible with the target character in the current round of iteration, the numerical range (maximum value, minimum value) of each attribute data of the characters in the fifth character set can be used to effectively identify the compatible numerical range of the target character determined by the result of this round of battle, thereby improving the determination speed of the third numerical interval.

[0080] When performing reinforcement learning, the specific process can be as follows:

[0081] The terminal device calls the initial AI model to control the non-player character to interact with each character in the first character set to obtain the sample data, and then performs reinforcement learning on the initial AI model based on the sample data. In an exemplary solution, taking the game scene as an example, the sample data may also include the game scene in the current battle situation and the status information of the game characters of the two opposing sides. In an exemplary solution, the server calls the initial game AI model to control NPC1 to fight against game character A, which is currently in game scene A, where NPC1 can use skill set A, game character A can use skill set B, NPC1's health is 80%, game character A's health is 50%, and so on.

[0082] It is understandable that the sample data includes the current game situation information and the label corresponding to the sample data. The game situation information can be the game situation information that any NPC can obtain, which can include at least one of the following, but not limited to: the NPC's position information, health information, skill information, distance information between the NPC and other game characters, obstacle information, time information, and the NPC's score information. Optionally, the time information can be the duration of the battle, etc., which is not limited in this application. Optionally, the game situation information also includes at least one of the following, but not limited to: the opponent's position information, health information, score information, skill information, etc. that the NPC can obtain. It should be understood that in the battle scene between the NPC and the player's game character or other NPCs, the NPC involved in the game situation information refers to the game character on the machine side. The other game characters involved in the game situation information can be the game character on the player side or other NPCs. The label corresponding to the sample data can be the action information output by the game AI model. The actions involved in the NPC or any fighting game AI model involved in this application include, but are not limited to: moving left, moving right, moving up, attacking, jumping, blocking, etc.

[0083] In an exemplary scenario, for each game process between two game characters, the process of the server obtaining sample data can be as follows: in the initial game situation, assuming that the game situation information obtained by NPC1 is S1, and the game situation information obtained by game character A is s1, S1 is input into the initial game AI model to obtain the action probability distribution of NPC1. At this time, based on the action probability distribution, game character A performs action a1. Similarly, s1 is input into the initial game AI model to obtain the action probability distribution of game character A. At this time, based on the action probability distribution, game character A performs action b1; then enter the next game situation, at this time NPC1 performs action a2, game character A performs action b2, and so on, until the two game characters decide the winner. Assuming that NPC1 ultimately wins after n battles, its corresponding samples are: (S1, a1, 1), (S2, a2, 1) ... (Sn, an, 1), and the samples corresponding to game character A are: (s1, b1, 0), (s2, b2, 0) ... (sn, bn, 0). Ultimately, these samples can constitute multiple sample data of the initial game AI model as described above. Among them, the 1 is used to indicate that NPC1 wins the battle, and the 0 is used to indicate that NPC1 loses the battle.

[0084] It is understandable that the losing party's sample data may contain a significant amount of information that could have significantly prevented the loss. Therefore, the terminal device may select at least one sample from the sample data corresponding to game character A for mutation. The terminal device may randomly select at least one sample corresponding to game character A or select at least one sample corresponding to game character A according to a specific rule. In short, this application does not restrict how the terminal device selects at least one sample corresponding to game character A. For example, the terminal device may randomly select 50% of the samples corresponding to game character A for mutation. The terminal device may adjust the second action probability distribution corresponding to the samples corresponding to game character A based on the second game situation information corresponding to the samples corresponding to game character A to obtain a third action probability distribution. For example, when game character A is attacked, the terminal device determines that game character A can choose to jump or block. If the jump or block probabilities obtained based on the action probability distribution in the current situation are not high, the terminal device may mutate the jump or block probabilities, such as increasing the jump or block probabilities. Although these mutated samples cannot guarantee that game character A will win, they can achieve a better result than failure. Therefore, the terminal device can mark the winning rate of game character A as between victory and failure, recorded as 2. Based on this, the server can form sample data with the samples corresponding to NPC1, the samples corresponding to game character A, and the samples generated by mutating the samples corresponding to game character A to train the initial game AI model.

[0085] 203. Randomly sample the plurality of attribute data in the second numerical range to generate a training character set.

[0086] The terminal device randomly samples the multiple attribute data in the second numerical interval to generate the training game character set. For example, assuming that the multiple attribute data and the numerical interval can be set as follows: attack power and its corresponding second numerical interval are (40, 90), defense power and its corresponding second numerical interval are (45, 98), and health and its corresponding second numerical interval are (50, 98); then the random sampling in the second numerical interval to generate the training game characters can be as follows: game character 1 (attack power: 50, defense power: 80, health: 85), game character 2 (attack power: 45, defense power: 98, health: 85), game character 3 (attack power: 55, defense power: 89, health: 90), game character 4 (attack power: 52, defense power: 81, health: 90), game character 5 (attack power: 65, defense power: 78, health: 90).

[0087] 204. Perform reinforcement learning training on the initial AI model using the training role set to obtain a target AI model.

[0088] In this embodiment, the process of the terminal device using the training role set to perform reinforcement learning training on the initial AI model is the same as in the above step 202, and the details will not be repeated here.

[0089] Optionally, in order to enrich the style of non-player characters, the terminal device can also set different reward functions for different non-player characters when performing reinforcement learning on the initial AI model, thereby training AI models with different style labels. The style label is used to identify the way the target character interacts with the opponent character. The style is used to allow the target character to simulate the user's emotions, making the user feel more involved. Different style labels are used to instruct the initial AI model to learn different interaction methods. The interaction methods guided by different style labels will be different. When the initial AI model controls the target character to interact in a way that conforms to the style label, it will receive a higher reward, which will be reflected through the reward function.

[0090] In an exemplary scenario, the terminal device can use the training role set and the first reward function of the initial AI model to perform reinforcement learning training on the initial AI model to obtain a first target AI model, and the first reward function is used to train the first style label of the first target AI model; use the training role set and the second reward function of the initial AI model to perform reinforcement learning training on the initial AI model to obtain a second target AI model, and the second reward function is used to train the second style label of the second target AI model. The first target AI model and the second target AI model serve as the target AI model, and the first target AI model and the second target AI model have different style labels.

[0091] For example, taking the game scene as an example, the first target AI model is the first target game AI model, and the second target AI model is the second target game AI model. At the same time, when training the first target game AI model, the reward for the game AI model to control the NPC to perform the "dodge" action is set to be the highest, then the style label of the trained first target game AI model can be "cautious"; or when training the second target game AI model, the reward for the game AI model to control the NPC to perform the "attack" action is set to be the highest, then the style label of the trained second target game AI model can be "aggressive".

[0092] Taking the AI ​​customer service scenario as an example, the first target AI model is the first target customer service AI model, and the second target AI model is the second target customer service AI model. At the same time, when training the first target customer service AI model, the reward for the customer service AI model to control the NPC to perform the "polite language" action is set to be the highest, then the style label of the trained first target customer service AI model can be "polite"; or when training the second target customer service AI model, the reward for the customer service AI model to control the NPC to perform the "popular science language" action is set to be the highest, then the style label of the trained second target customer service AI model can be "rational".

[0093] The first target AI model and the second target AI model obtained through training with different style labels can respectively control the target character to interact with the opponent character in different ways, effectively improving the interactive experience.

[0094] After training AI models with different style labels, the style indicators corresponding to the style labels can be used to measure whether the trained target AI model meets the model requirements (understandably, the model requirements can be set to style consistency, etc.). The specific operations can be as follows:

[0095] In an exemplary scheme, the above-mentioned first target game AI model and the second target game AI model are used as examples for explanation: the style index of the first target game AI model is set to the first style index, and the style index of the second target game AI model is set to the second style index (in this application, the style index can be understood as a quantifiable behavior index, for example, the style index can be set to the number of times the NPC performs the "dodge" action and the number of times the NPC performs the first "attack" action, or there are other settings, which are not limited here); then call the first target game AI to control the NPC to interact with each game character in the training game character set to obtain a first style index set (also called a first style index distribution), and call the first target game AI to control the NPC to interact with each game character in the test game character set to obtain The second style indicator set (also called the second style indicator distribution); at the same time, the second target game AI is also called to control the NPC to interact with each game character in the training game character set to obtain a third style indicator set (also called the third style indicator distribution), and the second target game AI is called to control the NPC to interact with each game character in the test game character set to obtain a fourth style indicator set (also called the fourth style indicator distribution); then, if the first target game AI model is measured, the measurement processing is performed according to the first style indicator set and the second style indicator set to obtain a first measurement value; at the same time, the measurement processing is performed according to the first style indicator set and the fourth style indicator set to obtain a second measurement value; then, based on the first measurement value and the second measurement value, it is determined whether the first target game AI model meets the model requirements. Similarly, if the second target game AI model is measured, the above solution can also be used, and the specific details are not limited here.

[0096] In this embodiment, the terminal device calculates the metric value based on each style indicator in the following manner:

[0097] In one possible implementation, to quantify the style indicators, the terminal device needs to normalize the style indicators output from the training game character set and the test game character set. Specifically, the first style indicator set and the third style indicator set are treated as a style indicator distribution and then normalized to obtain a standardized training style indicator distribution. Simultaneously, the second style indicator set and the third style indicator set are treated as a style indicator distribution and then normalized to obtain a standardized testing style indicator distribution. The metric value is then calculated based on the standardized training style indicator distribution and the standardized testing style indicator distribution.

[0098] In this embodiment, the standardization method can be to calculate the mean and variance, and then standardize the style index based on the mean and variance; deviation standardization can also be used. In other words, as long as the style index can be standardized, the specific method is not limited here.

[0099] Similarly, the method for calculating the measurement value based on the standardized training style indicator distribution and the standardized test style indicator distribution can be to use the average distance method to calculate the inter-class distance, or to calculate the KL divergence, or to calculate the total variation distance, or to calculate the Wasserstein distance. The specific method is not limited here, as long as the style indicator can be measured.

[0100] In an exemplary solution, select game AI model A and game AI model B with different style labels; conduct N games on the training set and test set respectively; assume that the style index is α, and record the style index of game AI model A in the i-th game on the training set as The indicator for the jth game on the test set is It can be obtained that the distribution of game AI model A and game AI model B in the training set and test set with style index α is and Calculate the mean and variance of the two distributions, respectively (μ, σ 2 ) and (μ′, σ′ 2 ), and make a distribution analysis of the index in the training set Standardize and get the distribution of the indicator on the standardized training set Similarly, the distribution of this indicator on the standardized validation set is obtained For the style index α, the inter-class distance of the average distance method is used to quantify the generalization (that is, the inter-class distance is used as the metric value). The inter-class distance between the game AI model A on the training set and the game AI model A on the test set is Similarly, the inter-class distance between game AI model A on the training set and game AI model B on the test set is D aB In practical applications, the smaller the distance between classes, the closer the distributions between the two are. For game AI model A, if D aA <D aX This indicates that Game AI Model A's style metrics on both the test and training sets are closest to those of other Game AI Models, demonstrating good generalization on the test set. Similarly, the generalization of the style metrics of all other Game AI Models A can be analyzed.

[0101] The following describes the application of the target game AI model using a game scenario as an example in the embodiment of the present application, as shown in FIG2b :

[0102] 301. Obtain current game situation information, where the game situation information includes status information of an NPC, status information of a game character competing against the NPC, and game scene information, wherein the NPC is a game character controlled by the target game AI model.

[0103] In this embodiment, the game situation information may be information available to any NPC, and may include at least one of the following, but is not limited to: the NPC's location information, health information, skill information, distance information between the NPC and other game characters, obstacle information, time information, and the NPC's score information. Optionally, the time information may include, but is not limited to, game duration, etc. Optionally, the game situation information may also include at least one of the following, but is not limited to, the opponent's location information, health information, score information, skill information, etc. available to the NPC. It should be understood that in a combat scenario, the NPC referred to in the game situation information refers to the game character on the machine side. The other game characters referred to in the game situation information may be the game character on the player side. The label corresponding to the sample data may be action information output by the game AI model. Actions associated with the NPC, or actions associated with any combat game AI model described in this application, include, but are not limited to, moving left, moving right, moving up, attacking, jumping, blocking, etc.

[0104] 302. Input the current game situation information into the target game AI model to obtain the NPC's action probability distribution.

[0105] The server inputs the current game situation information into the target game AI model to obtain the NPC's action probability distribution. In this embodiment, the action probability distribution output by the target game AI model is the probability distribution of each NPC action. For example, the NPC's corresponding actions include moving left, moving right, moving up, attacking, jumping, and blocking, and their corresponding execution probabilities are 0.2, 0.3, 0.6, 0.8, 0.3, and 0.2, respectively. In this case, the action probability distribution is (0.2, 0.3, 0.6, 0.8, 0.3, 0.2).

[0106] 303. Control the action of the NPC according to the action probability distribution.

[0107] In this embodiment, the action probability distribution output by the target game AI model is used by the client where the NPC is located to determine the action to be performed in the current game situation. For example, the action with the highest probability according to the action probability distribution is performed.

[0108] The application of the above-mentioned target game AI model is explained below with a specific application architecture, as shown in Figure 2c: it includes a game client and a server corresponding to the game, wherein the target game AI model is deployed in the server. In actual application, the game client obtains game description information, NPC status information and status information of the NPC's opposing game character, and sends the above information to the server; then the server calls the target game AI model to analyze the above information to obtain the NPC's decision, and sends the decision to the game client. The game client controls the NPC to perform corresponding operations according to the decision.

[0109] The model training device in the present application is described in detail below. Please refer to FIG3 , which is a schematic diagram of an embodiment of the model training device in the present application. The model training device 20 includes:

[0110] An acquisition module 201 is configured to acquire an initial artificial intelligence (AI) model and a first numerical range, wherein the initial AI model is used to control a target character, and the first numerical range is a numerical range of multiple attribute data corresponding to an opponent character, wherein the opponent character is used to interact with the target character;

[0111] Processing module 202 is configured to call the initial AI model to determine a second numerical interval from the first numerical interval, where the second numerical interval is adapted to the target character; and randomly sample the plurality of attribute data in the second numerical interval to generate a training character set;

[0112] The training module 203 is used to perform reinforcement learning training on the initial AI model using the training role set to obtain a target AI model.

[0113] In an embodiment of the present application, a model training device is provided. Using the above device, reinforcement learning is used to train the NPC model in the interactive scene, which can reduce the manual maintenance of the model and achieve rapid model training. At the same time, during the training process, the attribute data of the combat characters is numerically limited to ensure the validity of the sample data during the NPC model training process. At the same time, random sampling is performed within the selected numerical range to generate the combat characters, making the sample data more extensive, which can improve the generalization of the NPC AI model.

[0114] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0115] The processing module 202 is configured to randomly sample the plurality of attribute data in the first numerical range to generate a first character set;

[0116] Performing reinforcement learning training on the initial AI model using the first character set to obtain a first AI model;

[0117] Randomly sampling the plurality of attribute data in the first numerical interval to generate a second character set;

[0118] Invoking the first AI model to control the non-player character to interact with the second character set to generate a first evaluation result;

[0119] Calculating a first screening condition value of the first AI model according to the first evaluation result;

[0120] Determining a third value interval of the plurality of attribute data according to the first screening condition value and a preset screening threshold;

[0121] Randomly sampling the plurality of attribute data in the third numerical interval to generate a third character set;

[0122] Performing reinforcement learning training on the initial AI model using the third character set to obtain a second AI model;

[0123] Randomly sampling the plurality of attribute data in the third numerical interval to generate a fourth character set;

[0124] Invoking the second AI model to control the non-player character to interact with the fourth character set to generate a second evaluation result;

[0125] Calculating a second screening condition value of the second AI model according to the second evaluation result;

[0126] determining a fourth value interval of the plurality of attribute data according to the second screening condition value and the preset screening threshold;

[0127] The above operation is repeated until the numerical intervals of the plurality of attribute data reach a convergence condition, and the converged numerical interval is output as the second numerical interval.

[0128] In an embodiment of the present application, a model training device is provided. Using the device, the numerical range is limited by the evaluation results of the AI ​​model, which can exclude extreme sample data, thereby achieving numerical limits on the attribute data of the combat characters and ensuring the validity of the sample data during the NPC model training process.

[0129] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in an embodiment of the present application, the processing module 202 is configured to call the first AI model control non-player character to interact M times with N characters in the second character set to obtain N battle data sets, wherein each battle data set includes M groups of battle data, the battle data including but not limited to battle results and battle duration, each character in the second character set corresponds to a set of attribute data, and M and N are positive integers;

[0130] The N battle data sets are collected as the first evaluation result.

[0131] In an embodiment of the present application, a model training device is provided. Using the device, battle results and battle durations obtained through multiple interactions with characters are used as evaluation results in a screening process. This evaluation result can be used to limit the numerical range, thereby excluding extreme sample data, thereby achieving numerical limits on the attribute data of the combat characters and ensuring the validity of the sample data during the NPC model training process.

[0132] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0133] The processing module 202 is used to perform statistical processing on each of the N battle data sets in turn to obtain N groups of battle result parameters of the first AI model for the second character set, wherein the N groups of battle result parameters serve as the first filtering condition values.

[0134] In an embodiment of the present application, a model training device is provided. Using the device, multiple sets of data from the evaluation results are processed to obtain a set of comparable data, thereby simplifying data screening and facilitating the use of the evaluation results to limit the numerical range. This can eliminate extreme sample data, thereby achieving numerical limits on the attribute data of the combat characters and ensuring the validity of the sample data during the NPC model training process.

[0135] Optionally, based on the embodiment corresponding to FIG. 3 above, the battle result parameters include the battle win rate and the battle duration. In another embodiment of the model training device 20 provided in the embodiment of the present application,

[0136] The processing module 202 is configured to perform probability statistics on the battle results in the first battle data set to obtain the battle winning rate, wherein the first battle data set is included in the N battle data sets;

[0137] Averaging the battle times in the first battle data set to obtain the battle duration;

[0138] The N battle data sets are traversed in this manner to obtain N sets of victory rates and battle durations of the first AI model for the second character set.

[0139] In an embodiment of the present application, a model training device is provided. Using the above device, probability statistics are performed based on multiple battle results, which can conveniently obtain the winning rate of the current AI model when fighting against the opponent. At the same time, the battle time is averaged to obtain the average battle time of the current AI model and the opponent. This allows for more efficient data comparison, and thus facilitates limiting the numerical range based on the evaluation results. This can exclude extreme sample data, thereby achieving numerical limitation of the attribute data of the fighting character and ensuring the validity of the sample data during the NPC model training process.

[0140] Optionally, based on the embodiment corresponding to FIG. 3 , in another embodiment of the model training device 20 provided in an embodiment of the present application, the processing module 202 is configured to traverse the N groups of battle result parameters and determine X groups of battle result parameters from the N groups of battle result parameters according to the preset screening threshold, where X is a positive integer;

[0141] Determining a fifth character set corresponding to the X sets of battle result parameters, where the fifth character set is included in the second character set;

[0142] The third numerical range is determined according to a plurality of attribute data corresponding to each character in the fifth character set.

[0143] In an embodiment of the present application, a model training device is provided. Using the device, a matching character is identified by screening conditions. Different matching characters correspond to different attribute data. Therefore, if a matching character meets the requirements, its corresponding attribute data also meets the requirements. The numerical range of the initial attribute data is then narrowed based on the attribute data. This effectively limits the numerical range, thereby eliminating extreme sample data and ensuring the validity of the sample data during the NPC model training process.

[0144] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in an embodiment of the present application, the processing module 202 is configured to determine the maximum and minimum values ​​of each attribute data corresponding to each role in the fifth role set;

[0145] The third numerical range is determined according to the maximum value and the minimum value.

[0146] In an embodiment of the present application, a model training device is provided. Using the above device, the numerical interval is determined by the maximum and minimum values ​​of the attribute data, thereby gradually limiting the numerical interval, which can effectively and accurately limit the numerical interval.

[0147] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0148] The training module 203 is configured to perform reinforcement learning training on the initial AI model using the training character set and a first reward function of the initial AI model to obtain a first target AI model, wherein the first reward function is used to train a first style label of the first target AI model;

[0149] The initial AI model is subjected to reinforcement learning training using the training character set and the second reward function of the initial AI model to obtain a second target AI model. The second reward function is used to train the second style label of the second target AI model. The first target AI model and the second target AI model serve as the target AI model, and the first target AI model and the second target AI model have different style labels.

[0150] In an embodiment of the present application, a model training device is provided. Using the above device, different AI models with different styles can be trained by setting different reward functions for different reinforcement learning schemes, thereby enriching the behavior of NPCs in the game and increasing playability.

[0151] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0152] The processing module 202 is configured to call the first target AI model to control the non-player character to interact with the training character set to obtain a first style index set, and call the first target AI model to control the non-player character to interact with the test character set to obtain a second style index set, where the first style index set and the second style index set are quantitative indices corresponding to the first style label;

[0153] Invoking the second target AI model to control the non-player character to interact with the training character set to obtain a third style index set, and invoking the second target AI model to control the non-player character to interact with the test character set to obtain a fourth style index set, the third style index set and the fourth style index set being quantitative indices corresponding to the second style label;

[0154] Obtaining a first metric value of the first target AI model on the training character set and the test character set according to the first style indicator set and the second style indicator set;

[0155] Obtaining, according to the first style indicator set and the fourth style indicator set, a second metric value of the first target AI model on the training character set and a second metric value of the second target AI model on the test character set;

[0156] When the first metric value is less than the first threshold and the first metric value is less than the second metric value, it is determined that the first target AI model meets the model requirements.

[0157] In an embodiment of the present application, a model training device is provided. Using this device, style labels are indicated using style indicators, and the distribution of style indicators obtained from the training and test sets is obtained. The distribution of style indicators of the AI ​​models in the two datasets is compared. When the style indicators are close, the AI ​​model is determined to meet the training requirements. The distribution of style indicators of AI models under different style labels is also compared. This allows for a more accurate evaluation of the generalization of the AI ​​model through comparison of these two dimensions.

[0158] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0159] A processing module 202 is configured to perform standardization processing on the first style indicator set and the second style indicator set to obtain a first training style indicator set and a first testing style indicator set;

[0160] The first metric value is obtained according to the first training style indicator set and the first testing style indicator set.

[0161] In an embodiment of the present application, a model training device is provided. By using the device, the style indicators are normalized and then the metric values ​​are calculated, which can improve the accuracy of the metric values ​​and thus achieve better style indicator comparison.

[0162] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0163] The processing module 202 is configured to measure each style indicator in the first training style indicator set and the first testing style indicator set using an average distance method to obtain the first inter-class distance, which serves as the first metric value.

[0164] In an embodiment of the present application, a model training device is provided. Using the device, the style indicators in two data sets are measured using inter-class distance, thereby better comparing the style indicators and determining the generalization of the AI ​​model.

[0165] Optionally, based on the embodiment corresponding to FIG. 3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0166] The processing module 202 is configured to obtain a third metric value of the second target AI model on the training character set and the test character set based on the third style indicator set and the fourth style indicator set;

[0167] Obtaining, based on the third style indicator set and the second style indicator set, a fourth metric value of the second target AI model on the training character set and the first target AI model on the test character set;

[0168] When the third metric value is less than the first threshold and the third metric value is less than the fourth metric value, it is determined that the second target AI model meets the model requirements.

[0169] In an embodiment of the present application, a model training device is provided. Using this device, style labels are indicated using style indicators, and the distribution of style indicators obtained from the training and test sets is obtained. The distribution of style indicators of the AI ​​models in the two datasets is compared. When the style indicators are close, the AI ​​model is determined to meet the training requirements. The distribution of style indicators of AI models under different style labels is also compared. This allows for a more accurate evaluation of the generalization of the AI ​​model through comparison of these two dimensions.

[0170] Optionally, based on the embodiment corresponding to FIG3 above, in another embodiment of the model training device 20 provided in the embodiment of the present application, the target AI model is applied to a game scene, and the target character is a game character.

[0171] In an embodiment of the present application, a model training device is provided. Using the above device, reinforcement learning is used to train the NPC model in the game scene, which can reduce the manual maintenance of the model and achieve rapid model training. During the training process, numerical limits are set for the attribute data of the game characters to ensure the validity of the sample data during the NPC model training process. Random sampling is performed within the selected numerical range to generate the game characters, making the sample data more extensive, thereby improving the generalization of the NPC game AI model.

[0172] The game model training device provided by the present application can be used for a server, please refer to Figure 4, Figure 4 is a schematic diagram of a server structure provided by an embodiment of the present application, the server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and a memory 332, one or more storage media 330 (for example, one or more massive storage devices) for storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0173] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0174] The steps executed by the server in the above embodiment may be based on the server structure shown in FIG4 .

[0175] The game model training device provided by this application can be used for terminal devices. Please refer to Figure 5. For the sake of convenience, only the parts related to the embodiment of this application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of this application. In the embodiment of this application, the terminal device is a smartphone as an example for explanation:

[0176] FIG5 is a block diagram showing a partial structure of a smartphone related to a terminal device provided in an embodiment of the present application. Referring to FIG5 , the smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. It will be understood by those skilled in the art that the smartphone structure shown in FIG5 does not constitute a limitation on the smartphone, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0177] The following is a detailed introduction to the various components of a smartphone in conjunction with Figure 5:

[0178] The RF circuit 410 may be used for sending and receiving information or receiving and sending signals during a call.

[0179] The memory 420 may be used to store software programs and modules. The processor 480 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 420 .

[0180] The input unit 430 may be configured to receive input digital or character information and generate key signal input related to user settings and function control of the smartphone.

[0181] The display unit 440 may be used to display information input by the user or information provided to the user and various menus of the smartphone.

[0182] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors.

[0183] The audio circuit 460 , the speaker 461 , and the microphone 462 may provide an audio interface between the user and the smartphone.

[0184] The processor 480 is the control center of the smartphone, connecting the various parts of the entire smartphone using various interfaces and lines. It executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 420 and calling data stored in the memory 420.

[0185] The smartphone also includes a power source 490 (eg, a battery) for supplying power to various components.

[0186] The steps executed by the terminal device in the above embodiment may be based on the terminal device structure shown in FIG5 .

[0187] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0188] An embodiment of the present application also provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0189] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an artificial intelligence model, the method being executed by a terminal device, the method comprising: Acquire an initial artificial intelligence AI model and a first numerical range, wherein the initial AI model is used to control a target character, and the first numerical range is a numerical value range of multiple attribute data corresponding to an opponent character, and the opponent character is used to interact with the target character; Calling the initial AI model to determine a second numerical interval from the first numerical interval, where the second numerical interval is adapted to the target character; Randomly sampling the plurality of attribute data in the second numerical range to generate a training character set; The initial AI model is trained through reinforcement learning using the training role set to obtain a target AI model.

2. According to the method of claim 1, calling the initial AI model to determine the second numerical interval from the first numerical interval comprises: Randomly sampling the plurality of attribute data in the first numerical interval to generate a first role set; Performing reinforcement learning training on the initial AI model using the first role set to obtain a first AI model; Randomly sampling the plurality of attribute data in the first numerical interval to generate a second role set; Calling the first AI model to control the non-player character to interact with the second character set to generate a first evaluation result; Calculate a first screening condition value of the first AI model according to the first evaluation result; Determine a third value interval of the plurality of attribute data according to the first screening condition value and a preset screening threshold; Randomly sampling the plurality of attribute data in the third numerical interval to generate a third role set; Performing reinforcement learning training on the initial AI model using the third role set to obtain a second AI model; Randomly sampling the plurality of attribute data in the third numerical interval to generate a fourth role set; Calling the second AI model to control the non-player character to interact with the fourth character set to generate a second evaluation result; Calculate a second screening condition value of the second AI model according to the second evaluation result; Determine a fourth value interval of the plurality of attribute data according to the second screening condition value and the preset screening threshold; The above operation is repeated until the numerical intervals of the plurality of attribute data reach a convergence condition, and the converged numerical interval is output as the second numerical interval.

3. According to the method of claim 2, calling the first AI model to control the non-player character to interact with the second character set to generate the first evaluation result comprises: Calling the first AI model to control the non-player character to interact with N characters in the second character set M times respectively to obtain N battle data sets, wherein each battle data set includes M groups of battle data, and the battle data includes but is not limited to battle results and battle durations, each character in the second character set corresponds to a group of attribute data, and M and N are positive integers; The N battle data sets are combined as the first evaluation result.

4. According to the method of claim 3, calculating the first screening condition value of the first AI model according to the first evaluation result comprises: Statistical processing is performed on each of the N battle data sets in turn to obtain N groups of battle result parameters of the first AI model for the second character set, wherein the N groups of battle result parameters serve as the first filtering condition values.

5. The method according to claim 4, wherein the battle result parameters include a battle win rate and a battle duration, and wherein the statistical processing is performed on each of the N battle data sets in sequence to obtain N sets of battle result parameters of the first AI model for the second character set, including: Performing probability statistics on the battle results in a first battle data set to obtain the battle winning rate, wherein the first battle data set is included in the N battle data sets; Averaging the battle times in the first battle data set to obtain the battle duration; In this way, the N battle data sets are traversed to obtain N sets of victory rates and battle durations of the first AI model against the second character set.

6. The method according to claim 4, wherein determining the third value interval of the plurality of attribute data according to the first screening condition value and a preset screening threshold value comprises: Traversing the N groups of battle result parameters, and determining X groups of battle result parameters from the N groups of battle result parameters according to the preset screening threshold, where X is a positive integer; Determine a fifth character set corresponding to the X groups of battle result parameters, where the fifth character set is included in the second character set; The third numerical range is determined according to a plurality of attribute data corresponding to each role in the fifth role set.

7. The method according to claim 6, wherein determining the third numerical range according to the plurality of attribute data corresponding to the fifth role set comprises: Determine a maximum value and a minimum value of each attribute data corresponding to each role in the fifth role set; The third numerical interval is determined according to the maximum value and the minimum value.

8. According to the method according to any one of claims 1 to 7, the using the training role set to perform reinforcement learning training on the initial AI model to obtain the target AI model comprises: Performing reinforcement learning training on the initial AI model using the training character set and a first reward function of the initial AI model to obtain a first target AI model, wherein the first reward function is used to train a first style label of the first target AI model; The initial AI model is subjected to reinforcement learning training using the training role set and the second reward function of the initial AI model to obtain a second target AI model, wherein the second reward function is used to train a second style label of the second target AI model, and the first target AI model and the second target AI model are used as the target AI models, and the first target AI model and the second target AI model have different style labels.

9. The method according to claim 8, further comprising: Calling the first target AI model to control the non-player character to interact with the training character set to obtain a first style index set, and calling the first target AI model to control the non-player character to interact with the test character set to obtain a second style index set, wherein the first style index set and the second style index set are quantitative indicators corresponding to the first style label, and the test character set is generated by randomly sampling the multiple attribute data in the second numerical range; The second target AI model is called to control the non-player character to interact with the training character set to obtain a third style index set, and the second target AI model is called to control the non-player character to interact with the test character set to obtain a fourth style index set, wherein the third style index set and the fourth style index set are the second style label pair. Quantitative indicators of the application; Obtaining a first metric value of the first target AI model on the training character set and the test character set according to the first style indicator set and the second style indicator set; Obtaining second metric values ​​of the first target AI model on the training character set and the second target AI model on the test character set according to the first style indicator set and the fourth style indicator set; When the first metric value is less than a first threshold and the first metric value is less than the second metric value, it is determined that the first target AI model meets the model requirements.

10. The method according to claim 9, wherein obtaining a first metric value of the first target AI model on the training role set and the test role set according to the first style indicator set and the second style indicator set comprises: Standardizing the first style indicator set and the second style indicator set to obtain a first training style indicator set and a first testing style indicator set; The first metric value is obtained according to the first training style indicator set and the first testing style indicator set.

11. The method according to claim 10, wherein obtaining the first metric value according to the first training style indicator set and the first test style indicator set comprises: The average distance method is used to measure each style indicator in the first training style indicator set and the first test style indicator set to obtain the first inter-class distance, and the first inter-class distance is used as the first measurement value.

12. The method according to any one of claims 9 to 11, further comprising: Obtaining a third metric value of the second target AI model on the training character set and the test character set according to the third style indicator set and the fourth style indicator set; Obtaining, according to the third style indicator set and the second style indicator set, a fourth metric value of the second target AI model on the training role set and the first target AI model on the test role set; When the third metric value is less than the first threshold and the third metric value is less than the fourth metric value, it is determined that the second target AI model meets the model requirements.

13. According to the method according to any one of claims 1 to 11, the target AI model is applied to a game scene, and the target character is a game character.

14. A model training device, comprising: An acquisition module, used to acquire an initial artificial intelligence AI model and a first numerical range, where the first numerical range is a numerical value range of multiple attribute data corresponding to the opponent character of the initial AI model; A processing module, configured to call the initial AI model to determine a second numerical interval from the first numerical interval; Randomly sampling the plurality of attribute data in the second numerical range to generate a training character set; A training module is used to perform reinforcement learning training on the initial AI model using the training role set to obtain a target AI model.

15. A computer device comprising: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the method according to any one of claims 1 to 13 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

16. A storage medium for storing a computer program for executing the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 13.

Images