Game character behavior control method and device, and electronic device

Abstract

The application provides a game role behavior control method and device and electronic equipment; wherein the method comprises: obtaining current state data of a target game; inputting the current state data into a pre-trained machine learning model to obtain a first output result; wherein the first output result comprises probabilities corresponding to a plurality of preset behavior operations; obtaining a specified number of historical output results of the machine learning model; inputting the first output result and the historical output result into a pre-trained conditional random field model to obtain a target behavior operation. In this way, the machine learning model and the conditional random field model are combined, the machine learning model and the conditional random field model can both establish a relationship between a state and a behavior operation, the conditional random field model can also establish a time sequence relationship between behavior operations, the intelligent degree of the game AI can be high, and meanwhile, the model structure is simple and easy to converge, and has strong applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of games, and in particular to a game character behavior control method and device and electronic equipment. BACKGROUND

[0002] Game AI (Artificial Intelligence) is also called a virtual player. A game AI model is trained by using a machine learning technology, and a game character is controlled by using the game AI model, so that the behavior of the game character is similar to the behavior of a game character controlled by a real player. The behaviors of the game character before and after are usually related, and therefore, the game AI model is trained by using a time sequence model, so that the intelligent performance of the game character controlled by the game AI model is better.

[0003] In related technologies, an RNN (Recurrent Neural Network) and an LSTM (Long Short-Term Memory) model can establish a relationship between input data and output data to a certain extent, but it is difficult to establish the time sequence continuity of the output data, which affects the intelligent degree of the model during operation. An LSTM and a CRF (Conditional Random Field) loss function can establish the time sequence relationship between the input data and the output data and between the output data, but the model complexity is high, it is difficult to converge effectively, and the application of the model is poor. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a game character behavior control method and device and electronic equipment, so that the intelligent degree of the game AI is high, the model structure is simple and easy to converge, and the application is strong.

[0005] In a first aspect, the present application provides a game character behavior control method, which comprises: obtaining current state data of a target game; inputting the current state data into a pre-trained machine learning model to obtain a first output result; wherein the first output result comprises probabilities corresponding to a plurality of preset behavior operations; obtaining a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; inputting the first output result and the historical output results into a pre-trained conditional random field model to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of a target game character.

[0006] In a second aspect, an embodiment of the present application provides a behavior control apparatus for a game character, the apparatus comprising: a data acquisition module configured to acquire current state data of a target game; a first input module configured to input the current state data into a pre-trained machine learning model to obtain a first output result; wherein the first output result comprises probabilities corresponding to a plurality of preset behavior operations; a result acquisition module configured to acquire a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; and a second input module configured to input the first output result and the historical output results into a pre-trained conditional random field model to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control a behavior of a target game character.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory storing machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the behavior control method for a game character.

[0008] In a fourth aspect, an embodiment of the present application provides a machine readable storage medium, the machine readable storage medium storing machine executable instructions, and the machine executable instructions, when invoked and executed by a processor, cause the processor to implement the behavior control method for a game character.

[0009] The embodiments of the present application have the following beneficial effects:

[0010] The behavior control method, apparatus and electronic device for a game character acquire current state data of a target game; input the current state data into a pre-trained machine learning model to obtain a first output result; wherein the first output result comprises probabilities corresponding to a plurality of preset behavior operations; acquire a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; and input the first output result and the historical output results into a pre-trained conditional random field model to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control a behavior of a target game character. In this way, the machine learning model and the conditional random field model are combined, the machine learning model and the conditional random field model can both establish a relationship between a state and a behavior operation, and the conditional random field model can also establish a time sequence relationship between behavior operations, so that the degree of intelligence of a game AI can be high; meanwhile, the model structure is simple and easy to converge, and has strong applicability.

[0011] Other features and advantages of the present application will be set forth in the descriptions that follow, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structures particularly pointed out in the description, claims and drawings.

[0012] In order to make the above objectives, features and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and make a detailed description with the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0014] Figure 1 A flow chart of a behavior control method of a game character provided by an embodiment of the present application;

[0015] Figure 2 A schematic diagram of a training method of a machine learning model provided by an embodiment of the present application;

[0016] Figure 3 A schematic diagram of a training method of a conditional random field model provided by an embodiment of the present application;

[0017] Figure 4 A structural schematic diagram of a behavior control device of a game character provided by an embodiment of the present application;

[0018] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0020] Game AI is an essential element in many games, and a game AI with intelligent performance can bring a better game experience to players. For example, in a Moba (Multiplayer Online Battle Arena) game, there is usually a game AI that controls a hero as a player does, and the game character controlled by the game AI competes with or cooperates with the game character controlled by the player. More and more games use machine learning technology to train game AI with higher strength and more intelligent performance.

[0021] Most game AI models use non-time sequence models for modeling, and in addition, the technology using time sequence models usually uses RNN (recurrent neural network) or LSTM (long short-term memory network). Among them, the non-time sequence model cannot meet the modeling of the continuity feature of the model output value. The RNN and LSTM networks can establish the relationship between the observation and the state in time sequence to a certain extent through the time sequence memory redundancy reservation of the intermediate layer of the network, but the disadvantage is the lack of modeling of the state time sequence continuity. The LSTM plus CRF loss function method has higher complexity of the model than the LSTM and CRF two itself, and is not convenient for effective convergence during training.

[0022] Based on the above problems, the embodiment of the application provides a game character behavior control method, device and electronic equipment, which can be applied to the control of game AI in various games, for example, the control of game AI in a Moba game.

[0023] In order to facilitate the understanding of the present embodiment, first, a game character behavior control method disclosed by the embodiment of the application is introduced in detail, as shown in the method can be applied to a server, a cloud server or a terminal device, etc. The method comprises the following steps: Figure 1

[0024] Step S102, obtaining current state data of a target game;

[0025] The target game usually includes a plurality of game characters, part of which are controlled by real players, and the target game character in the present embodiment is a game AI, that is, the target behavior operation is output by the game character behavior control method provided by the present embodiment, and then the target game character is controlled to execute the target behavior operation.

[0026] The current state data of the target game may, for example, include environment state data of a game match, such as match environment, match progress, number of characters of each party, state, etc. The current state data may also include character state data of each game character in the game match, such as the position, blood volume, attack power and resistance ability of the character. In one way, the current state data may also be the above-mentioned character state data of the target game character to be controlled. ​

[0027] Step S104, input the current state data into the machine learning model trained in advance to obtain a first output result; wherein the first output result includes probabilities corresponding to a plurality of preset behavior operations;

[0028] The machine learning model can be a neural network model or other types of machine learning models. The machine learning model can be trained by training data, for example, during the running of the target game, the character state data of the real player-controlled game character is collected, and the behavior operation triggered by the player under the character state data; of course, the training data can also include environment state data of the environment where the game character is located; through these training data, the machine learning model can learn the way the real player makes decisions based on the current state data of the game character. The machine learning model can be a time series model or a non-time series model; the machine learning model can use a relatively simple model structure to achieve.

[0029] The above-mentioned preset behavior operation can include multiple types, such as movement operation, movement in a specific direction, attack operation, use of specified attack skill, use of specified defense skill, etc. In the above-mentioned first output result, the probability corresponding to each preset behavior operation is included, wherein the behavior operation with the maximum probability value is the behavior operation that the machine learning model considers to be the most matched under the current state data.

[0030] Step S106, obtain a specified number of historical output results of the machine learning model; wherein the historical output result is obtained based on the historical state data of the target game;

[0031] It can be understood that as time changes, the state of the target game will change, so before the aforementioned current state data is generated, there will usually be at least one historical state data; that is, the historical state data is generated before the aforementioned current state data. Each historical state data is input into the machine learning model to output a corresponding historical output result. In actual implementation, the historical output result corresponding to the historical state data before the current state data can be obtained in reverse order of time sequence from the aforementioned current state data.

[0032] For example, the specified number is three, then three historical output results are obtained, the three historical output results correspond to three historical state data, the historical state data is adjacent in time sequence, and the last historical state sequence is adjacent to the current state data. That is, the historical state data and the current state sequence are continuous in time.

[0033] Step S108, input the first output result and the historical output result into the conditional random field model trained in advance to obtain a target behavior operation; wherein the target behavior operation is included in a plurality of preset behavior operations, and the target behavior operation is used to control the behavior of the target game character.

[0034] The conditional random field model also needs to be trained in advance. Unlike the foregoing machine learning model, the conditional random field model needs to be trained by using time-continuous state data and a behavior operation corresponding to the last state data, that is, the conditional random field model needs to learn, based on the time-continuous state data, to output a final behavior operation. The target behavior is not only related to the latest state data, but also related to the historical state data.

[0035] The conditional random field model is a discriminative probabilistic model and a kind of random field. The conditional random field model is a conditional probability distribution model P(Y|X), which represents a Markov random field of another group of output random variables Y given a group of input random variables X, that is, the conditional random field model assumes that the output random variables form a Markov random field. The conditional random field model can be regarded as a generalization of the maximum entropy Markov model to the labeling problem.

[0036] Like the Markov random field, the conditional random field model is a graph model without direction. In the conditional random field, the distribution of the random variable Y is a conditional probability, and the given observation value is the random variable X. In principle, the graph model layout of the conditional random field model can be given arbitrarily. The commonly used layout is a chain architecture. The chain architecture has high-efficiency algorithms for training, inference, or decoding. The conditional random field model is a typical discriminative model, and its joint probability can be written in the form of the product of several potential functions. The most commonly used is a linear chain conditional random field.

[0037] In actual application, the foregoing first output result corresponding to the current state data and the historical output result corresponding to the historical state data are jointly input into the conditional random field model to output a final target behavior operation. The target behavior operation belongs to one or more of the foregoing plurality of preset behavior operations. After obtaining the target behavior operation, the corresponding target game character in the target game can be controlled to execute the target behavior operation, thereby achieving the purpose of controlling the game character by the game AI.

[0038] The behavior control method of the game character, current state data of a target game is obtained; the current state data is input into a machine learning model that is pre-trained to obtain a first output result; wherein the first output result includes probabilities corresponding to a plurality of preset behavior operations; a specified number of historical output results of the machine learning model are obtained; wherein the historical output results are obtained based on historical state data of the target game; the first output result and the historical output results are input into a conditional random field model that is pre-trained to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of the target game character. In this way, the machine learning model and the conditional random field model are combined, the machine learning model and the conditional random field model can both establish a relationship between the state and the behavior operation, the conditional random field model can also establish a time sequence relationship between the behavior operations, and the intelligence level of the game AI can be high; at the same time, the model structure is simple and easy to converge, and has strong applicability.

[0039] In a specific implementation, the current state data includes one or more of position data, health data, physical attack strength data, spell attack strength data, physical defense data, and spell defense data of the target game character.

[0040] The position data can be the position of the target game character in a game scene, which can be expressed using a three-dimensional space coordinate table of the game scene. The health data is the blood volume of the target game character, which can specifically include the total blood volume and the current blood volume of the target game character; the physical attack is a direct attack without using skills; the physical attack strength data indicates the ability of the target game character to physically attack; the spell attack is an attack using skills, and the spell attack strength data indicates the ability of the target game character to spell attack; the physical defense is a defense against physical attacks, thereby reducing the damage to the target game character caused by physical attacks; the physical defense strength data indicates the defense ability of the target game character against physical attacks; the spell defense is a defense against spell attacks, thereby reducing the damage to the target game character caused by spell attacks; and the spell defense strength data indicates the defense ability of the target game character against spell attacks.

[0041] In actual implementation, the corresponding state data can be collected at each game frame, and the state data corresponding to the last game frame is the current state data in time sequence. During model running, the current state data can be set as a data vector with a length of K, where K represents the data type of the current state data; for example, if the current state data includes position data, total blood volume, current blood volume, physical attack strength data, spell attack strength data, physical defense strength data, and spell defense strength data, then the value of K is 7.

[0042] AsFigure 2 As shown in the above machine learning model is trained by the following way:

[0043] Step S202, obtaining first training data; wherein the first training data includes: state data in a plurality of game frames of a specified game role in a target game, and behavior operation of the specified game role corresponding to the state data;

[0044] The specified game role is usually a game role controlled by a real player; as the target game progresses, the game frames are arranged in time sequence. For each game frame, the state data of the specified game role in the game frame can be collected, as well as the behavior operation of the specified game role under the state data; the behavior operation is triggered by the real player through the terminal device. The behavior operation is usually one or more of the aforementioned plurality of preset behavior operations.

[0045] In a specific implementation, assuming that there are N game frames, each game frame corresponds to a state data, and the state data contains K states, then the dimension of the state data is K; the data dimension of the state data in the aforementioned first training data is N*K; in the first training data, the data Y of the behavior operation corresponding to the state data, the dimension of Y is N*1; assuming that the type of preset behavior operation is A, then the dimension of Y is N*A, wherein the vector of A consists of 0 and 1; for example, the A vector in the data Y of the behavior operation corresponding to the state data is represented as [0, 0, 1], which represents that the behavior operation corresponding to the state data is the third behavior operation.

[0046] Step S204, inputting the state data in the first training data into an initial model of the machine learning model to obtain a second output result;

[0047] The initial model of the machine learning model is initialized with initial model parameters; the state data in the first training data is input into the initial model as a training sample of the machine learning model, and the initial model outputs the second output result.

[0048] Step S206, taking the behavior operation in the first training data as label information, training the model parameters of the initial model based on the second output result and the label information until the initial model converges, and obtaining the trained machine learning model.

[0049] The initial model is preset with a loss function, the loss value between the label information and the second output result is calculated through the loss function, based on the loss value, the model parameters of the initial model are adjusted using gradient descent method or other parameter adjustment method, then the aforementioned step S204 and step S206 are continued to be executed until the initial model converges, and the trained machine learning model is obtained.

[0050] As Figure 3As shown, the conditional random field model is trained in the following manner:

[0051] In step S302, second training data is obtained; the second training data includes state data in a plurality of game frames of a specified game character in a target game and behavior operations of the specified game character corresponding to the state data.

[0052] The second training data can be the same as, partially the same as, or completely different from the first training data. In one mode, the second training data is the first training data, which is used to train the machine learning model and the conditional random field model.

[0053] The specified game character is usually a game character controlled by a real player; the game frames are arranged in time sequence as the target game progresses. For each game frame, state data of the specified game character in the game frame and behavior operations of the specified game character under the state data can be collected; the behavior operations are triggered by the real player through a terminal device. The behavior operations are usually one or more of the plurality of preset behavior operations.

[0054] In step S304, the state data in the second training data is input into the pre-trained machine learning model to obtain a third output result.

[0055] It should be noted that the machine learning model in this step is a trained model, and the third output result corresponding to each state data is obtained after the state data is input into the machine learning model; the third output result includes a probability vector corresponding to each state data, i.e., a probability value of each preset behavior operation corresponding to the state data; for example, assuming that there are three preset behavior operations, for state data A, the probability vector is [0.1, 0.88, 0.02], which includes the probability of each preset behavior operation; the behavior operation corresponding to the probability value 0.88 is the behavior operation that the machine learning model considers most suitable for the target game character to perform under the state data A.

[0056] In actual implementation, the machine learning model can include a softmax function, which outputs the probability vector corresponding to each state data.

[0057] In step S306, a plurality of sequence samples are generated based on the third output result; the sequence sample is composed of a preset number of third output results.

[0058] Since the state data is arranged in the order of game frames, the third data set results corresponding to each state data are also arranged in the order of game frames, and in each sequence sample, a plurality of continuous third output results can be included; at least part of the third output results in different sequence samples are different.

[0059] In a specific implementation, the state data in the second training data is arranged in the order of game frames, and the third output results corresponding to the state data are arranged in the order of the state data; at least one target result is determined from the third output results; the target result and a preset number of previous results before the target result are determined as a sequence sample; wherein the preset number of previous results and the target result are arranged in the order of the state data; the preset number of previous results are adjacent to the target result; the target results corresponding to different sequence samples are different.

[0060] For example, the conditional random field model presets the length of the inputtable sequence sample as L, L represents the number of third output results included in the sequence sample; Y represents the third output result; the i-th third output result is determined as the target result, denoted as Y'(i); all third output results starting from Y'(i-L+1) and ending at Y'(i) are concatenated in the order to form a sequence sample, denoted as Y^(i). Wherein, Y'(i-L+1) is the start, and the third output results from Y'(i-1) are all previous results. When i changes, each i can obtain a sequence sample Y^(i), and thus a plurality of sequence samples can be obtained.

[0061] In step S308, the behavior operation in the second training data is taken as state information, the sequence sample is taken as an observation sequence, and the model parameters of the conditional random field model are trained based on the state information and the observation sequence until the conditional random field model converges, and the trained conditional random field model is obtained.

[0062] The conditional random field model is preset with a loss function, and the loss value between the state information and the observation sequence is calculated through the loss function. Based on the loss value, the model parameters of the conditional random field model are adjusted using the gradient descent method or other parameter adjustment methods, and then step S308 is continuously executed multiple times until the conditional random field model converges, and the trained conditional random field model is obtained.

[0063] Based on the conditional random field model obtained by the above training, the target behavior operation is obtained in the following manner. The first output result is input into a preset data queue; the data queue has a specified length, at least one historical output result is saved in the data queue, and the data in the data queue is arranged in the order of input; if the data input into the data queue exceeds the specified length, the data first input into the data queue is deleted; if the data in the data queue reaches the specified length, the data in the data queue is input into the conditional random field model pre-trained to output the target behavior operation.

[0064] The above data queue can be a circular queue, denoted as a circular queue Q; the length of the circular queue is L, data is stored from one end of the circular queue, and the data is arranged in the order of storage; when the length of the data in the circular queue reaches L, if data continues to be stored in the circular queue, the data first stored in the other end of the circular queue is deleted, and the same is true for subsequent data storage; in the subsequent data storage process, the circular queue always saves data with a length of L.

[0065] In actual implementation, the length of the above data queue is the same as the length of the sequence sample that can be input into the conditional random field model, both of which are L; L is the number of output results saved in the data queue.

[0066] In the initial state, the data in the data queue may not reach the specified length; in this case, if the data in the data queue does not reach the specified length, the behavior operation corresponding to the maximum probability value in the first output result is determined as the target behavior operation. Since the data in the data queue does not reach the specified length, the data cannot be input into the conditional random field model to form a data with a length required by the conditional random field model; in this case, the target behavior operation is directly determined based on the first output result output by the machine learning model, that is, the first output result includes a probability corresponding to each preset behavior operation, and the behavior operation with the maximum probability value is determined as the target behavior operation.

[0067] If the data in the data queue reaches the specified length, the data in the data queue is input into the conditional random field model pre-trained to output the behavior operation label corresponding to each historical output result in the data queue and the behavior operation label corresponding to the first output result; the behavior operation indicated by the behavior operation label corresponding to the first output result is determined as the target behavior operation.

[0068] If the data in the data queue reaches the specified length, the historical output result saved in the data queue is the specified number of output results before the first output result. When the first output result is constantly updated, the historical output result in the data queue is also constantly updated. Therefore, the first output result and the corresponding historical output result are jointly input into the conditional random field model to obtain the target behavior operation, which not only refers to the first output result, but also refers to the historical output result. Since the first output result is output by the machine learning model based on the current state data, and the historical output result is output by the machine learning model based on the historical state data, the target behavior operation not only refers to the current state data of the target game, but also refers to the historical state data of the target game.

[0069] The above-mentioned behavior control method of the game character is realized by the machine learning model and the conditional random field model, establishes the relationship between the state and the behavior operation, and the conditional random field model can also establish the time sequence relationship between the behavior operations, so that the intelligent degree of the game AI is relatively high; meanwhile, the model structure is simple and easy to converge, and has strong applicability. For the game AI in the MOBA game, by using the machine learning model and the conditional random field model, the continuity modeling of the MOBA game in the operation time sequence scene can be realized under the premise of ensuring that the model is easy to converge.

[0070] Corresponding to the above-mentioned method embodiment, referring to Figure 4 The behavior control device of the game character shown in the structural schematic view comprises:

[0071] The data acquisition module 40 is configured to acquire the current state data of the target game.

[0072] The first input module 42 is configured to input the current state data into the pre-trained machine learning model to obtain a first output result. The first output result comprises probabilities corresponding to a plurality of preset behavior operations.

[0073] The result acquisition module 44 is configured to acquire a specified number of historical output results of the machine learning model. The historical output results are obtained based on the historical state data of the target game.

[0074] The second input module 46 is configured to input the first output result and the historical output result into the pre-trained conditional random field model to obtain a target behavior operation. The target behavior operation is contained in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of the target game character.

[0075] The behavior control device of the game character obtains current state data of a target game; inputs the current state data into a machine learning model that is pre-trained and completed to obtain a first output result; wherein the first output result includes probabilities corresponding to a plurality of preset behavior operations; obtains a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; inputs the first output result and the historical output results into a conditional random field model that is pre-trained and completed to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of the target game character. In this way, the machine learning model and the conditional random field model are combined, the machine learning model and the conditional random field model can both establish a relationship between a state and a behavior operation, the conditional random field model can also establish a time sequence relationship between behavior operations, and the intelligent degree of the game AI can be high. At the same time, the model structure is simple and easy to converge, and has strong applicability.

[0076] The current state data includes one or more of position data, life value data, physical attack strength data, spell attack strength data, physical defense data, and spell defense data of the target game character.

[0077] The device further includes a first training module configured to: obtain first training data; wherein the first training data includes state data in a plurality of game frames of a specified game character in the target game, and behavior operations of the specified game character corresponding to the state data; input the state data in the first training data into an initial model of the machine learning model to obtain a second output result; take the behavior operations in the first training data as label information, and train model parameters of the initial model based on the second output result and the label information until the initial model converges to obtain the machine learning model that is trained and completed.

[0078] The second input module is further configured to: input the first output result into a preset data queue; wherein the data queue has a specified length, at least one historical output result is saved in the data queue, data in the data queue is arranged in an input order, if data input into the data queue exceeds the specified length, data input into the data queue first is deleted; if the data in the data queue reaches the specified length, the data in the data queue is input into the conditional random field model that is pre-trained and completed to output the target behavior operation.

[0079] The second input module is further configured to: if the data in the data queue reaches a specified length, input the data in the data queue into the conditional random field model that has been pre-trained, output the action operation label corresponding to each historical output result in the data queue and the action operation label corresponding to the first output result, and determine the action operation indicated by the action operation label corresponding to the first output result as the target action operation.

[0080] The device further includes an operation determination module configured to: if the data in the data queue does not reach the specified length, determine the action operation corresponding to the maximum probability value in the first output result as the target action operation.

[0081] The device further includes a second training module configured to: obtain second training data, wherein the second training data includes state data in a plurality of game frames of a target game of a specified game role and the action operation of the specified game role corresponding to the state data; input the state data in the second training data into the machine learning model that has been pre-trained to obtain a third output result; generate a plurality of sequence samples based on the third output result, wherein each sequence sample is composed of a preset number of third output results; take the action operation in the second training data as state information and take the sequence sample as an observation sequence, and train the model parameters of the conditional random field model based on the state information and the observation sequence until the conditional random field model converges to obtain the conditional random field model that has been trained.

[0082] The second training module is further configured to: determine at least one target result from the third output result; determine the target result and a preset number of previous results before the target result as a sequence sample, wherein the preset number of previous results and the target result are arranged in the order of the state data, the preset number of previous results are adjacent to the target result, and different sequence samples correspond to different target results.

[0083] The embodiment also provides an electronic device including a processor and a memory, the memory storing machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the behavior control method of the game role. The electronic device can be a server or a terminal device.

[0084] Referring to Figure 5 As shown in the figure, the electronic device includes a processor 100 and a memory 101, the memory 101 storing machine executable instructions capable of being executed by the processor 100, and the processor 100 executes the machine executable instructions to implement the behavior control method of the game role.

[0085] Further, Figure 5The electronic device also includes a bus 102 and a communication interface 103, the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102.

[0086] The memory 101 can include a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0087] The processor 100 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor 100 or instructions in the form of software. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is mature in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101, and combines the hardware to complete the steps of the method of the above embodiment.

[0088] The processor in the electronic device can implement the following operations in the behavior control method of the game character by executing machine executable instructions.

[0089] obtain current state data of the target game; input the current state data into a machine learning model that is pre-trained to obtain a first output result; wherein the first output result includes probabilities corresponding to a plurality of preset behavior operations; obtain a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; input the first output result and the historical output results into a conditional random field model that is pre-trained to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of the target game character.

[0090] The current state data includes one or more of position data, health data, physical attack strength data, spell attack strength data, physical defense data, and spell defense data of the target game character.

[0091] The machine learning model is trained in the following manner: obtain first training data; wherein the first training data includes state data in a plurality of game frames of a specified game character in the target game, and behavior operations of the specified game character corresponding to the state data; input the state data in the first training data into an initial model of the machine learning model to obtain a second output result; use the behavior operations in the first training data as label information, and train model parameters of the initial model based on the second output result and the label information until the initial model converges to obtain the trained machine learning model.

[0092] input the first output result into a preset data queue; wherein the data queue has a specified length, at least one historical output result is saved in the data queue, and data in the data queue is arranged in the order of input; if the data input into the data queue exceeds the specified length, the data first input into the data queue is deleted; if the data in the data queue reaches the specified length, the data in the data queue is input into the conditional random field model that is pre-trained to output the target behavior operation.

[0093] If the data in the data queue reaches the specified length, the data in the data queue is input into the conditional random field model that is pre-trained to output behavior operation labels corresponding to each historical output result in the data queue and a behavior operation label corresponding to the first output result; and the behavior operation indicated by the behavior operation label corresponding to the first output result is determined as the target behavior operation.

[0094] If the data in the data queue does not reach the specified length, the action operation corresponding to the maximum probability value in the first output result is determined as the target action operation.

[0095] The conditional random field model is trained in the following manner: obtaining second training data; wherein the second training data comprises: state data in a plurality of game frames of a specified game role in a target game, and behavior operations of the specified game role corresponding to the state data; inputting the state data in the second training data into the machine learning model that has been pre-trained to obtain a third output result; generating a plurality of sequence samples based on the third output result; wherein a sequence sample is composed of a preset number of third output results; taking the behavior operations in the second training data as state information, and taking the sequence samples as observation sequences, training model parameters of the conditional random field model based on the state information and the observation sequences until the conditional random field model converges, to obtain a trained conditional random field model.

[0096] The state data in the second training data is arranged in the order of game frames, and the third output results corresponding to the state data are arranged in the order of the state data; at least one target result is determined from the third output results; the target result and a preset number of previous results before the target result are determined as a sequence sample; wherein the preset number of previous results and the target result are arranged in the order of the state data; the preset number of previous results are adjacent to the target result; different sequence samples correspond to different target results.

[0097] In the above manner, the machine learning model and the conditional random field model are combined, the machine learning model and the conditional random field model can both establish the relationship between the state and the behavior operation, the conditional random field model can also establish the time sequence relationship between the behavior operations, which can make the intelligent degree of the game AI higher; at the same time, the model structure is simple and easy to converge, and has strong applicability.

[0098] The embodiment also provides a machine-readable storage medium, which stores machine-executable instructions, and the machine-executable instructions cause a processor to implement the above-mentioned behavior control method of the game role when the machine-executable instructions are called and executed by the processor.

[0099] The machine-executable instructions stored in the above-mentioned machine-readable storage medium can implement the following operations in the above-mentioned behavior control method of the game role by executing the machine-executable instructions:

[0100] obtaining current state data of a target game; inputting the current state data into a pre-trained machine learning model to obtain a first output result; wherein the first output result includes probabilities corresponding to a plurality of preset behavior operations; obtaining a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; inputting the first output result and the historical output results into a pre-trained conditional random field model to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of a target game character.

[0101] The current state data includes one or more of position data, health data, physical attack strength data, spell attack strength data, physical defense data, and spell defense data of the target game character.

[0102] The machine learning model is trained in the following manner: obtaining first training data; wherein the first training data includes state data in a plurality of game frames of a specified game character in a target game, and behavior operations of the specified game character corresponding to the state data; inputting the state data in the first training data into an initial model of the machine learning model to obtain a second output result; training model parameters of the initial model based on the second output result and label information of the behavior operations in the first training data until the initial model converges, to obtain the trained machine learning model.

[0103] inputting the first output result into a preset data queue; wherein the data queue has a specified length, at least one historical output result is saved in the data queue, and data in the data queue is arranged in the order of input; if the data input into the data queue exceeds the specified length, the data input into the data queue first is deleted; if the data in the data queue reaches the specified length, the data in the data queue is input into the pre-trained conditional random field model to output the target behavior operation.

[0104] If the data in the data queue reaches the specified length, the data in the data queue is input into the pre-trained conditional random field model to output behavior operation labels corresponding to each historical output result in the data queue and a behavior operation label corresponding to the first output result; and the behavior operation indicated by the behavior operation label corresponding to the first output result is determined as the target behavior operation.

[0105] If the data in the data queue does not reach the specified length, the behavior operation corresponding to the maximum probability value in the first output result is determined as the target behavior operation.

[0106] The conditional random field model is trained by the following manner: obtaining second training data; wherein the second training data comprises: state data in a plurality of game frames of a specified game role in a target game, and behavior operation of the specified game role corresponding to the state data; inputting the state data in the second training data into the machine learning model which is pre-trained to obtain a third output result; generating a plurality of sequence samples based on the third output result; wherein the sequence sample is composed of a preset number of third output results; taking the behavior operation in the second training data as state information, taking the sequence sample as an observation sequence, and training model parameters of the conditional random field model based on the state information and the observation sequence until the conditional random field model converges, to obtain the trained conditional random field model.

[0107] The state data in the second training data is arranged in the order of game frames, and the third output result corresponding to the state data is arranged in the order of the state data; at least one target result is determined from the third output result; the target result and a preset number of previous results before the target result are determined as a sequence sample; wherein the preset number of previous results and the target result are arranged in the order of the state data; the preset number of previous results are adjacent to the target result; different sequence samples correspond to different target results.

[0108] In the above manner, the machine learning model and the conditional random field model are combined, the machine learning model and the conditional random field model can both establish the relationship between the state and the behavior operation, the conditional random field model can also establish the time sequence relationship between the behavior operations, which can make the intelligent degree of the game AI higher; meanwhile, the model structure is simple and easy to converge, and has strong applicability.

[0109] The game role behavior control method, device and computer program product of the electronic equipment provided by the embodiment of the application include a machine-readable storage medium storing program codes, the instructions included in the program codes can be used to execute the method described in the foregoing method embodiment, and the specific implementation can be referred to the method embodiment, which will not be described here.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0111] In addition, in the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, or can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0112] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a machine readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the prior art that essentially contributes to the present application or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0113] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0114] Finally, it should be noted that: the above embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A behavior control method of a game character, characterized by, The method comprises: obtaining current state data of a target game; inputting the current state data into a pre-trained machine learning model to obtain a first output result; wherein the first output result comprises probabilities corresponding to a plurality of preset behavior operations; obtaining a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; inputting the first output result and the historical output results into a pre-trained conditional random field model to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of a target game character; The conditional random field model is trained in the following way: obtaining second training data; wherein the second training data comprises state data in a plurality of game frames of a specified game character in the target game, and behavior operations of the specified game character corresponding to the state data; inputting the state data in the second training data into a pre-trained machine learning model to obtain a third output result; based on the third output result, generating a plurality of sequence samples; wherein the sequence sample is composed of a preset number of third output results; the behavior operation in the second training data is taken as state information, and the sequence sample is taken as an observation sequence; the model parameters of the conditional random field model are trained based on the state information and the observation sequence until the conditional random field model converges, and a trained conditional random field model is obtained.

2. The method of claim 1, wherein, The current state data comprises one or more of position data, health data, physical attack strength data, spell attack strength data, physical defense data, and spell defense data of the target game character.

3. The method of claim 1, wherein, The machine learning model is trained in the following way: obtaining first training data; wherein the first training data comprises state data in a plurality of game frames of a specified game character in the target game, and behavior operations of the specified game character corresponding to the state data; inputting the state data in the first training data into an initial model of the machine learning model to obtain a second output result; the behavior operation in the first training data is taken as label information, and the model parameters of the initial model are trained based on the second output result and the label information until the initial model converges, and a trained machine learning model is obtained.

4. The method of claim 1, wherein, The step of inputting the first output result and the historical output result into a pre-trained conditional random field model to obtain a target behavior operation comprises: inputting the first output result into a preset data queue; wherein the data queue has a specified length, at least one historical output result is saved in the data queue, and the data in the data queue is arranged in the order of input; if the data input into the data queue exceeds the specified length, the data first input into the data queue is deleted. If the data in the data queue reaches the specified length, input the data in the data queue into a conditional random field model pre-trained to completion, and output a target behavior operation.

5. The method of claim 4, wherein, If the data in the data queue reaches the specified length, input the data in the data queue into a conditional random field model pre-trained to completion, and output a target behavior operation. If the data in the data queue reaches the specified length, input the data in the data queue into a conditional random field model pre-trained to completion, and output a target behavior operation. If the data in the data queue reaches the specified length, input the data in the data queue into a conditional random field model pre-trained to completion, and output a target behavior operation.

6. The method of claim 4, wherein, The method further comprises: If the data in the data queue does not reach the specified length, determine the behavior operation corresponding to the maximum probability value in the first output result as the target behavior operation.

7. The method of claim 1, wherein, The state data in the second training data is arranged in the order of the game frames, and the third output result corresponding to the state data is arranged in the order of the state data. The step of generating a plurality of sequence samples based on the third output result comprises: Determining at least one target result from the third output result; Determining the target result and a preset number of previous results before the target result as a sequence sample; wherein the preset number of previous results and the target result are arranged in the order of the state data; the preset number of previous results are adjacent to the target result; different sequence samples correspond to different target results.

8. An apparatus for controlling behavior of a game character, characterized by: The device comprises: A data acquisition module for acquiring current state data of a target game; A first input module for inputting the current state data into a machine learning model pre-trained to completion to obtain a first output result; wherein the first output result includes probabilities corresponding to a plurality of preset behavior operations; A result acquisition module for acquiring a specified number of historical output results of the machine learning model; wherein the historical output results are obtained based on historical state data of the target game; A second input module for inputting the first output result and the historical output results into a conditional random field model pre-trained to completion to obtain a target behavior operation; wherein the target behavior operation is included in the plurality of preset behavior operations, and the target behavior operation is used to control the behavior of a target game character; A model training module for training the conditional random field model in the following manner: Acquiring second training data; wherein the second training data includes state data in a plurality of game frames of a specified game character in the target game, and behavior operations of the specified game character corresponding to the state data; Inputting the state data in the second training data into the machine learning model pre-trained to completion to obtain a third output result; generate a plurality of sequence samples based on the third output result; wherein the sequence sample is composed of a preset number of the third output result; take the behavior operation in the second training data as state information, take the sequence sample as an observation sequence, train model parameters of the conditional random field model based on the state information and the observation sequence, until the conditional random field model converges, and obtain the trained conditional random field model.

9. An electronic device, comprising: The game role behavior control method comprises the following steps: a processor and a memory are included, the memory stores machine executable instructions which can be executed by the processor, and the processor executes the machine executable instructions to implement the game role behavior control method in any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions, and the machine executable instructions cause the processor to implement the game role behavior control method in any one of claims 1-7 when the machine executable instructions are called and executed by the processor.

Images