Game strategy model training method and device, electronic equipment and storage medium

By obtaining and using different game reward function parameters to train the game strategy model, the problem that NPC design in the existing technology relies on a large amount of user data is solved, and the training of multiple styles of NPC in the game development stage is realized, reducing debugging time and Methods of computing resources.

CN118987627APending Publication Date: 2024-11-22NETEASE (HANGZHOU) NETWORK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310576024.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, the design of multi-style NPCs requires a large amount of player data, and the level of NPCs imitating learning training is difficult to exceed the level of human players, and it is impossible to effectively realize multiple styles of NPCs in the game development stage.

Method used

By obtaining several different game reward function parameters, the game strategy model is trained to adapt to different game reward function parameters, thereby training multiple NPCs of different styles without relying on user data with different styles.

Benefits of technology

It realizes that multiple NPCs of different styles are trained without relying on large amounts of user data, reducing the time and computing resources required for multi-style NPC debugging, and improving the adaptability and expressiveness of NPCs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118987627A_ABST
    Figure CN118987627A_ABST
Patent Text Reader

Abstract

The invention provides a game strategy model training method and device, electronic equipment and a storage medium. The method comprises the steps that a plurality of different game reward function parameters are acquired; respectively taking a plurality of different game reward function parameters as a part of input data of a to-be-trained game strategy model, jointly performing first training on the to-be-trained game strategy model until the to-be-trained game strategy model converges, and obtaining a game strategy model after the first training, the first trained game strategy model is used for controlling the non-player character to interact with the player character. The game reward function parameters participate in training of the game strategy model, the correlation between the game strategy model parameters and the game reward function parameters is high, the game strategy model can adapt to different game reward function parameters, and on this basis, the game reward function parameters can be optimized. By adjusting the parameters of the game reward function, various game strategy models of different styles can be obtained through training, and the method does not depend on user data of different styles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of electronic computers and Internet technology, and in particular to a method, device, electronic device and storage medium for training a game strategy model. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] In electronic games, it is usually necessary to configure a variety of different styles of NPCs (non-player characters). NPCs interact with users based on the game strategy model and provide services to users, including but not limited to communicating with users, competing with users, or assisting users in their actions.

[0004] In the related art, NPCs are usually trained based on user data, so that NPCs can imitate different types of users to obtain NPCs of different styles. However, this solution requires a large amount of user data, and the style and performance of NPCs are greatly limited by the quantity and quality of user data. Summary of the invention

[0005] In view of this, the purpose of the present disclosure is to propose a training method, device, electronic device and storage medium for a game strategy model, which at least to a certain extent solves one of the technical problems in the related art.

[0006] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a method for training a game strategy model, including:

[0007] Get several different game reward function parameters;

[0008] The several different game reward function parameters are respectively used as part of the input data of the game strategy model to be trained, and the game strategy model to be trained is jointly subjected to a first training until the game strategy model to be trained converges to obtain a first trained game strategy model, wherein the first trained game strategy model is used to control a non-player character to interact with a player character.

[0009] Based on the same inventive concept, the second aspect of the exemplary embodiment of the present disclosure provides a training device for a game strategy model, including:

[0010] A game reward function parameter acquisition module, configured to acquire a number of different game reward function parameters;

[0011] The game strategy model training module is configured to use the several different game reward function parameters as part of the input data of the game strategy model to be trained, and jointly perform a first training on the game strategy model to be trained until the game strategy model to be trained converges to obtain a first trained game strategy model, wherein the first trained game strategy model is used to control a non-player character to interact with a player character.

[0012] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0013] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0014] From the above, it can be seen that the training method, device, electronic device and storage medium of the game strategy model provided by the embodiment of the present disclosure include: obtaining several different game reward function parameters; using several different game reward function parameters as part of the input data of the game strategy model to be trained, and jointly performing a first training on the game strategy model to be trained until the game strategy model to be trained converges to obtain a first trained game strategy model, wherein the first trained game strategy model is used to control non-player characters to interact with player characters. Among them, since the game reward function parameters participate in the training of the game strategy model, the parameters of the game strategy model have a high correlation with the game reward function parameters, and the game strategy model can adapt to different game reward function parameters. On this basis, adjusting the game reward function parameters can train game strategy models of various styles, which does not rely on user data of different styles. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A schematic diagram of an application scenario of the method for training a game strategy model provided in an embodiment of the present disclosure;

[0017] Figure 2A flowchart of a method for training a game strategy model provided by an embodiment of the present disclosure;

[0018] Figure 3 Another flowchart of a method for training a game strategy model provided by an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of a structure of a training device for a game strategy model provided by an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0022] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0023] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0024] In the related art, the design of NPCs with different styles usually needs to rely on a large amount of player data.

[0025] Specifically, the imitation learning algorithm based on player data is a method for implementing multi-style NPCs. The approach is: first, collect a large amount of player data, that is, the game environment status faced by the player at different times in a game and the action data taken by the player. Secondly, classify the collected player data. The classification method may be to use an unsupervised clustering algorithm, or to divide the player according to various indicators in a game. Use the data of each category to train a strategy model with the imitation learning algorithm, and you can get NPCs of various styles.

[0026] However, the inventors of the present disclosure have found that the above method is based on the premise that developers can obtain a large amount of player data, which is not possible in the game development stage. In addition, the level of NPCs trained by imitation learning is limited by the level of the players whose data is collected, and it is impossible to train NPCs that exceed the level of human players. Therefore, the multi-style NPC solution based on player data requires a large amount of player data, which is usually not possible for games in the development stage, and the level of NPCs trained by imitation learning is difficult to exceed the level of sampled players.

[0027] In order to solve the above problems, the present disclosure provides a training scheme for a game strategy model, which specifically includes: obtaining several different game reward function parameters; using several different game reward function parameters as part of the input data of the game strategy model to be trained, and jointly performing a first training on the game strategy model to be trained until the game strategy model to be trained converges, and obtaining a first trained game strategy model, wherein the first trained game strategy model is used to control non-player characters to interact with player characters. Among them, since the game reward function parameters participate in the training of the game strategy model, the parameters of the game strategy model have a high correlation with the game reward function parameters, and the game strategy model can adapt to different game reward function parameters. On this basis, adjusting the game reward function parameters can train game strategy models of various styles of games, which does not rely on user data of different styles.

[0028] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0029] refer to Figure 1 , which is a schematic diagram of an application scenario of the training method of the game strategy model provided by the embodiment of the present disclosure. The application scenario includes a terminal device 101, a server 102 and a data storage system 103. Among them, the terminal device 101, the server 102 and the data storage system 103 can be connected through a wired or wireless communication network. The terminal device 101 includes but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (personal digital assistant, PDA) or other electronic devices capable of realizing the above functions. The server 102 and the data storage system 103 can both be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms.

[0030] The server 102 is used to provide a training service of a game strategy model to a user of the terminal device 101. The terminal device 101 is installed with a client that communicates with the server 102. The user can perform relevant configuration operations on the game reward function parameters through the client, including but not limited to constructing, selecting and / or adjusting the game reward function parameters. The client sends the game reward function parameters to the server 102, and the server 102 obtains several different game reward function parameters; the several different game reward function parameters are respectively used as part of the input data of the game strategy model to be trained, and the game strategy model to be trained is jointly trained for the first time until the game strategy model to be trained converges to obtain a first trained game strategy model, wherein the first trained game strategy model is used to control a non-player character to interact with a player character. Based on the game strategy model obtained by the first training, the game strategy model can be trained again based on different game reward function parameters to train game strategy models of different styles.

[0031] The data storage system 103 stores training data, each of which includes game reward function parameters and game state data, as well as game actions corresponding to the game reward function parameters and game state data. The server 102 can train the game strategy model based on the training data, so that the game strategy model can output game actions according to the input game reward function parameters and game state data. The source of the training data includes but is not limited to an existing database, data crawled from the Internet, or data uploaded by the user when using the client. When the accuracy of the game strategy model output meets certain requirements, the server 102 can provide the user with a game strategy model training service based on the game strategy model. At the same time, the server 102 can also continuously optimize the game strategy model based on the newly added training data.

[0032] Combine the following Figure 1 The training method of the game strategy model according to the exemplary embodiment of the present disclosure is described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0033] refer to Figure 2 , which is a flow chart of the training method of the game strategy model provided in the embodiment of the present disclosure.

[0034] The training method of the game strategy model includes the following steps:

[0035] Step S210, obtaining several different game reward function parameters.

[0036] As an example, the pre-built game reward function is R(s,a|ω), where ω={v 1 ,ω 2 ,ω 3 …} are the game reward function parameters, s is the game state in the game state space S, and a is the game action in the game action space A.

[0037] In some exemplary embodiments, the obtaining of several different game reward function parameters includes:

[0038] Obtaining preset game reward function parameters and preset selection probabilities of the preset game reward function parameters;

[0039] Execute the following method several times to obtain the several different game reward function parameters:

[0040] Determining whether to adopt the preset game reward function parameters based on the preset selection probability;

[0041] If the preset game reward function parameter is adopted, the preset game reward function parameter is used as one of the several different game reward function parameters;

[0042] If the preset game reward function parameters are not adopted, new game reward function parameters are obtained based on the preset joint probability distribution function, and the new game reward function parameters are used as one of the several different game reward function parameters.

[0043] In some exemplary embodiments, the preset game reward function parameters include several items;

[0044] The method further comprises:

[0045] The value range of each item in the preset game reward function parameter is configured to obtain the preset joint probability distribution function.

[0046] In some exemplary embodiments, the step of obtaining a new game reward function parameter based on a preset joint probability distribution function includes:

[0047] Random sampling is performed based on the preset joint probability distribution function to obtain the newly added game reward function parameters.

[0048] As an example, a set of preset game reward function parameters ω is pre-built 0 , and the preset selection probability p of the preset game reward function parameters 0 .

[0049] As an example, based on the preset game reward function parameters, the value range of each preset game reward function parameter is set. in, They represent the upper and lower bounds of the i-th preset game reward function parameter, and the value range contains the corresponding preset game reward function parameter, and the pre-built preset joint probability distribution function is obtained as D(ω 1 ,ω 2 ,ω 3 …).

[0050] As an example, before a game starts, it is determined whether to use the preset game reward function parameter ω in this game according to the preset selection probability of the preset game reward function parameter. 0 , if the preset game reward function parameter ω is used 0 , then the preset game reward function parameter ω 0 As the reward function parameter of this game; if the preset game reward function parameter ω is not used 0 , then according to the preset joint probability distribution function D(ω 1 ,ω 2 ,ω 3 …) Randomly obtain a set of game reward function parameters ω k ={ω k,1 ,ω k,2 ,ω k,3 …} as the reward function parameter for this game.

[0051] refer to Figure 3 In some exemplary embodiments, the method of using the several different game reward function parameters as part of the input data of the game strategy model to be trained, and jointly performing a first training on the game strategy model to be trained until the game strategy model to be trained converges to obtain the first trained game strategy model includes:

[0052] Step S310, obtaining first game state data from the game environment where the game strategy model is located;

[0053] Step S320: for each of the game reward function parameters, input the game reward function parameter and the first game state data into the game strategy model to be trained, and obtain the game action output by the game strategy model to be trained;

[0054] Step S330, executing the game action, and obtaining second game state data after executing the game action from the game environment, and obtaining a game reward according to the game reward function parameter, the game action and the second game state data;

[0055] Step S340: Update the parameters of the game strategy model according to the game reward function parameters, the game action, the second game state data and the game reward until the game strategy model to be trained converges to obtain a first trained game strategy model.

[0056] In some exemplary embodiments, the step of obtaining first game state data from a game environment in which the game strategy model is located includes:

[0057] The current game environment factors are obtained from the game environment, and the first game state data is obtained according to the current game environment factors and a pre-constructed game state space, wherein the game state space includes a plurality of game state data.

[0058] In some exemplary embodiments, the first game state data includes:

[0059] At least one of state data of the game environment, state data of the player character, and state data of the non-player character.

[0060] Optionally, first game state data is selected from a plurality of game state data in the game state space according to current game environment factors.

[0061] In some exemplary embodiments, the method further comprises:

[0062] All game environment factors related to the game strategy model are acquired from the game environment, and the game state space is constructed according to all the game environment factors.

[0063] Among them, all game environment factors that need to be considered when making NPC (game strategy model) decisions are obtained in advance from the game environment, and the game state space S is constructed based on these game environment factors, wherein the game state space S contains several game states s.

[0064] Optionally, a set of all obtained game environment factors is used as the game state space S.

[0065] In some exemplary embodiments, the method further comprises:

[0066] A game action space of the game strategy model is constructed, wherein the game action space includes a plurality of game actions, and different game actions represent different game action categories and different values ​​of the same game action category.

[0067] The game action space A of the NPC (game strategy model) is pre-constructed, wherein the game action space A includes a number of game actions a, and different game actions a represent different game action categories and different values ​​of the same game action category.

[0068] Optionally, the same game action category may correspond to different values.

[0069] Optionally, the same game action category may correspond to the same value range.

[0070] In some exemplary embodiments, the method further comprises:

[0071] The dimension of the input layer of the game strategy model is determined according to the dimension of the game state space and the target game reward function parameter, and the dimension of the output layer of the game strategy model is determined according to the dimension of the game action space to construct the game strategy model.

[0072] According to the dimensions of the game state space, game action space, and game reward function parameters, the dimensions of the input and output of the game strategy model are determined, the game strategy model π(s,ω|θ) is constructed, and the network parameters θ of the game strategy model are randomly initialized. 0 .

[0073] Among them, the input dimension of the game strategy model is the sum of the dimension of the game state space and the dimension of the game reward function parameter, and the output dimension is the dimension of the game action space.

[0074] As an example, assuming that the dimension of the game state space is |S| and the dimension of the game reward function parameter is |ω|, the dimension of the input of the game strategy model is the sum of the two, that is, |S|+|ω|.

[0075] Optionally, the number of network layers in the model and the type of network used in each layer depend on the input type of the state space.

[0076] Among them, in the game, each time a decision is made, observe the current game state s t , after normalization and other data processing, it is consistent with the game reward function parameter ω of this game k Input them into the game strategy model together to get the game action a t .

[0077] Among them, the game reward is r t =R(s t ,a t |ω k ).

[0078] In some exemplary embodiments, updating the parameters of the game strategy model according to the target game reward function parameters, the game action, the second game state, and the game reward until the game strategy model to be trained converges to obtain a first trained game strategy model includes:

[0079] splicing the target game reward function parameter, the game action, the second game state and the game reward into an experience sample;

[0080] According to the experience samples, the loss function of the game strategy model is calculated based on the reinforcement learning algorithm, and the parameters of the game strategy model are updated based on the gradient descent algorithm until the game strategy model to be trained converges to obtain the first trained game strategy model.

[0081] Among them, the game reward function parameters, game actions, the second game state and the game reward are spliced ​​into an experience sample and stored.

[0082] Among them, the experience samples collected by interacting with the game environment are used to calculate the loss function of the game strategy model according to the reinforcement learning algorithm, and the gradient descent algorithm is used to update the parameters of the game strategy model until the game strategy model to be trained converges to obtain the first trained game strategy model.

[0083] In the present disclosure, the parameters of the reward function are used as input in the model training process to participate in the model training; wherein, the parameters of the reward function are not fixed, but there are multiple ones, so that the trained model can adapt to different reward function parameters; at this point, the first training of the model is completed, and the model has the ability to determine different decision-making styles according to specific reward parameters.

[0084] In some exemplary embodiments, the method further comprises:

[0085] Get the target game reward function parameters;

[0086] Using the target game reward function parameter as a part of the input data of the first trained game strategy model, performing a second training on the first trained game strategy model until the first trained game strategy model converges, thereby obtaining a second trained game strategy model, wherein the second trained game strategy model is used to control the non-player character to interact with the player character;

[0087] Among them, the second trained game strategy model obtained by performing the second training based on different target game reward function parameters can output different output data for the same input data.

[0088] Optionally, based on the game strategy model obtained by the first training, within the parameter selection range, select a set of game reward function parameters and the game state as the input of the game strategy model. Play several games, observe the performance of the NPC in the game and the statistical data of various indicators, and fine-tune the game reward function parameters according to the difference between the performance and data and the actual needs, and play several games again. Repeat the above steps until the required NPCs of various styles are obtained.

[0089] It should be noted that in the first training of the game strategy model, the first trained game strategy model is a model that can adapt to different game reward function parameters, that is, the first trained model can give more accurate results no matter which game reward function parameters are used. In the second training, based on a specific game reward function parameter, the first trained game strategy model is fine-tuned so that the output of the first trained game strategy model can be more targeted at the specific game reward function parameter. In this case, the first trained game strategy model is trained twice based on multiple specific game reward function parameters, and multiple game strategy models corresponding to multiple specific game reward function parameters can be obtained. In addition, since the game strategy models target different game reward function parameters, different actions will be output for the same input. In this case, these game strategy models can be said to have different styles.

[0090] From the above, it can be seen that in the training method of the game strategy model provided by the embodiment of the present disclosure, since the game reward function parameters participate in the training of the game strategy model, the parameters of the game strategy model are highly correlated with the game reward function parameters, and the game strategy model can adapt to different game reward function parameters. On this basis, adjusting the game reward function parameters can train NPCs of various different styles, which is independent of user data of different styles.

[0091] Specifically, the game strategy model can perceive the difference in long-term accumulated game rewards for decisions under different game reward function parameters at the beginning of the training process. Thanks to the fitting ability of the deep neural network in the game strategy model and the strong correlation between the game reward function parameters and performance, the game strategy model can quickly adapt to different game reward function parameters and learn to make different decisions under different game reward function parameters. After the game strategy model training is completed, you only need to adjust the input game reward function parameters to directly change the decision of the game strategy model. No further training is required to obtain NPCs with different performance styles. Compared with the solutions in the related art, the present disclosure can also significantly reduce the time and computing resources required for debugging multi-style NPCs.

[0092] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0093] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a training device for a game strategy model.

[0095] refer to Figure 4 , a training device for a game strategy model, comprising:

[0096] A game reward function parameter acquisition module 410 is configured to acquire a number of different game reward function parameters;

[0097] The game strategy model training module 420 is configured to use the several different game reward function parameters as part of the input data of the game strategy model to be trained, and jointly perform a first training on the game strategy model to be trained until the game strategy model to be trained converges, thereby obtaining a first trained game strategy model, wherein the first trained game strategy model is used to control a non-player character to interact with a player character.

[0098] In some exemplary embodiments, the game reward function parameter acquisition module 410 is specifically configured to:

[0099] Obtaining preset game reward function parameters and preset selection probabilities of the preset game reward function parameters;

[0100] Execute the following method several times to obtain the several different game reward function parameters:

[0101] Determining whether to adopt the preset game reward function parameters based on the preset selection probability;

[0102] If the preset game reward function parameter is adopted, the preset game reward function parameter is used as one of the several different game reward function parameters;

[0103] If the preset game reward function parameters are not adopted, new game reward function parameters are obtained based on the preset joint probability distribution function, and the new game reward function parameters are used as one of the several different game reward function parameters.

[0104] In some exemplary embodiments, the game reward function parameter acquisition module 410 is specifically configured to:

[0105] The value range of each item in the preset game reward function parameter is configured to obtain the preset joint probability distribution function.

[0106] In some exemplary embodiments, the game reward function parameter acquisition module 410 is specifically configured to:

[0107] Random sampling is performed based on the preset joint probability distribution function to obtain the newly added game reward function parameters.

[0108] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0109] Acquire first game state data from the game environment where the game strategy model is located;

[0110] For each of the game reward function parameters, input the game reward function parameter and the first game state data into the game strategy model to be trained, and obtain the game action output by the game strategy model to be trained;

[0111] Execute the game action, and obtain second game state data after executing the game action from the game environment, and obtain a game reward according to the game reward function parameter, the game action and the second game state data;

[0112] According to the game reward function parameters, the game action, the second game state data and the game reward, the parameters of the game strategy model are updated until the game strategy model to be trained converges to obtain a first trained game strategy model.

[0113] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0114] The current game environment factors are obtained from the game environment, and the first game state data is obtained according to the current game environment factors and a pre-constructed game state space, wherein the game state space includes a plurality of game state data.

[0115] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0116] All game environment factors related to the game strategy model are acquired from the game environment, and the game state space is constructed according to all the game environment factors.

[0117] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0118] A game action space of the game strategy model is constructed, wherein the game action space includes a plurality of game actions, and different game actions represent different game action categories and different values ​​of the same game action category.

[0119] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0120] The dimension of the input layer of the game strategy model is determined according to the dimension of the game state space and the game reward function parameter, and the dimension of the output layer of the game strategy model is determined according to the dimension of the game action space to construct the game strategy model.

[0121] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0122] splicing the game reward function parameter, the game action, the second game state and the game reward into an experience sample;

[0123] According to the experience samples, the loss function of the game strategy model is calculated based on the reinforcement learning algorithm, and the parameters of the game strategy model are updated based on the gradient descent algorithm until the game strategy model to be trained converges to obtain the first trained game strategy model.

[0124] In some exemplary embodiments, the first game state data includes:

[0125] At least one of state data of the game environment, state data of the player character, and state data of the non-player character.

[0126] In some exemplary embodiments, the game strategy model training module 420 is specifically configured to:

[0127] Get the target game reward function parameters;

[0128] Using the target game reward function parameter as a part of the input data of the first trained game strategy model, performing a second training on the first trained game strategy model until the first trained game strategy model converges, thereby obtaining a second trained game strategy model, wherein the second trained game strategy model is used to control the non-player character to interact with the player character;

[0129] Among them, the second trained game strategy model obtained by performing the second training based on different target game reward function parameters can output different output data for the same input data.

[0130] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0131] The device of the above embodiment is used to implement the training method of the corresponding game strategy model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0132] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the game strategy model described in any of the above embodiments is implemented.

[0133] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0134] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0135] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0136] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0137] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0138] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0139] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0140] The electronic device of the above embodiment is used to implement the training method of the corresponding game strategy model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0141] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the training method of the game strategy model as described in any of the above embodiments.

[0142] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0143] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the training method of the game strategy model as described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0144] It is known to those skilled in the art that the embodiments of the present disclosure may be implemented as a system, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some exemplary embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0145] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0146] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0148] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0150] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0151] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0152] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0153] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0154] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A method for training a game strategy model, characterized in that: include: Get several different game reward function parameters; The several different game reward function parameters are respectively used as part of the input data of the game strategy model to be trained, and the game strategy model to be trained is jointly subjected to a first training until the game strategy model to be trained converges to obtain a first trained game strategy model, wherein the first trained game strategy model is used to control a non-player character to interact with a player character.

2. The method according to claim 1, characterized in that The obtaining of several different game reward function parameters includes: Obtaining preset game reward function parameters and preset selection probabilities of the preset game reward function parameters; Execute the following method several times to obtain the several different game reward function parameters: Determining whether to adopt the preset game reward function parameters based on the preset selection probability; If the preset game reward function parameter is adopted, the preset game reward function parameter is used as one of the several different game reward function parameters; If the preset game reward function parameters are not adopted, new game reward function parameters are obtained based on the preset joint probability distribution function, and the new game reward function parameters are used as one of the several different game reward function parameters.

3. The method according to claim 2, characterized in that The preset game reward function parameters include several items; The method further comprises: The value range of each item in the preset game reward function parameter is configured to obtain the preset joint probability distribution function.

4. The method according to claim 2, characterized in that: The newly added game reward function parameters are obtained based on the preset joint probability distribution function, including: Random sampling is performed based on the preset joint probability distribution function to obtain the newly added game reward function parameters.

5. The method according to claim 1, characterized in that The method of using the plurality of different game reward function parameters as a part of the input data of the game strategy model to be trained respectively, and jointly performing a first training on the game strategy model to be trained until the game strategy model to be trained converges to obtain the first trained game strategy model comprises: Acquire first game state data from the game environment where the game strategy model is located; For each of the game reward function parameters, input the game reward function parameter and the first game state data into the game strategy model to be trained, and obtain the game action output by the game strategy model to be trained; Execute the game action, and obtain second game state data after executing the game action from the game environment, and obtain a game reward according to the game reward function parameter, the game action and the second game state data; According to the game reward function parameters, the game action, the second game state data and the game reward, the parameters of the game strategy model are updated until the game strategy model to be trained converges to obtain a first trained game strategy model.

6. The method according to claim 5, characterized in that The step of obtaining first game state data from the game environment where the game strategy model is located includes: The current game environment factors are obtained from the game environment, and the first game state data is obtained according to the current game environment factors and a pre-constructed game state space, wherein the game state space includes a plurality of game state data.

7. The method according to claim 6, characterized in that The method further comprises: All game environment factors related to the game strategy model are acquired from the game environment, and the game state space is constructed according to all the game environment factors.

8. The method according to claim 6, characterized in that The method further comprises: A game action space of the game strategy model is constructed, wherein the game action space includes a plurality of game actions, and different game actions represent different game action categories and different values ​​of the same game action category.

9. The method according to claim 8, characterized in that The method further comprises: The dimension of the input layer of the game strategy model is determined according to the dimension of the game state space and the game reward function parameter, and the dimension of the output layer of the game strategy model is determined according to the dimension of the game action space to construct the game strategy model.

10. The method according to claim 5, characterized in that The updating of the parameters of the game strategy model according to the game reward function parameter, the game action, the second game state and the game reward until the game strategy model to be trained converges to obtain a first trained game strategy model comprises: splicing the game reward function parameter, the game action, the second game state and the game reward into an experience sample; According to the experience samples, the loss function of the game strategy model is calculated based on the reinforcement learning algorithm, and the parameters of the game strategy model are updated based on the gradient descent algorithm until the game strategy model to be trained converges to obtain the first trained game strategy model.

11. The method according to claim 5, characterized in that The first game state data includes: At least one of state data of the game environment, state data of the player character, and state data of the non-player character.

12. The method according to claim 1, characterized in that The method further comprises: Get the target game reward function parameters; Using the target game reward function parameter as a part of the input data of the first trained game strategy model, performing a second training on the first trained game strategy model until the first trained game strategy model converges, thereby obtaining a second trained game strategy model, wherein the second trained game strategy model is used to control the non-player character to interact with the player character; Among them, the second trained game strategy model obtained by performing the second training based on different target game reward function parameters can output different output data for the same input data.

13. A training device for a game strategy model, characterized in that: include: A game reward function parameter acquisition module, configured to acquire a number of different game reward function parameters; The game strategy model training module is configured to use the several different game reward function parameters as part of the input data of the game strategy model to be trained, and jointly perform a first training on the game strategy model to be trained until the game strategy model to be trained converges to obtain a first trained game strategy model, wherein the first trained game strategy model is used to control a non-player character to interact with a player character.

14. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 12 is implemented.

15. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 12.