Virtual character behavior control method and device, electronic equipment and storage medium

Abstract

The invention provides a virtual character behavior control method and device, electronic equipment and a storage medium, and relates to the technical field of games, and the method comprises the steps: obtaining a visual game image of a game environment at the current moment and quantifiable game features associated with a virtual character in the game environment; determining a target game behavior to be executed by the virtual character based on the visual game image and the quantifiable game features through a target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained based on large language model training; and controlling the virtual character to execute the target game behavior in the game environment, so that the virtual character interacts with the game environment. According to the method, the personification effect stability can be remarkably improved, and the overall personification efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of game technology, and in particular to a method, device, electronic equipment and storage medium for controlling the behavior of a virtual character. Background Art

[0002] In order to achieve anthropomorphic strategy generation, it is usually necessary to carefully design the reward function of the reinforcement learning algorithm. Although carefully adjusting the reward function can solve the problem that some agents are not similar enough to human behavior, due to the subjectivity of anthropomorphism itself, it is often difficult to design a specific reward function to optimize agent behavior. Therefore, researchers usually use imitation learning methods that are not based on reward functions to generate anthropomorphic agent behavior. As an offline data optimization scheme, imitation learning only requires the historical behavior sequence of the expert to imitate the expert strategy, thereby achieving objective imitation of the expert behavior.

[0003] Although imitation learning achieves the learning of game behaviors from an objective perspective, the complexity and diversity of human data make it difficult for imitation learning to effectively imitate more detailed game behaviors. For this reason, most existing technologies will perform secondary optimization on the strategies learned by imitation learning from a subjective perspective. Reinforcement learning based on human feedback is a reinforcement learning algorithm that uses human subjective preferences to optimize the strategy of the model. Specifically, it first interacts the strategy learned by imitation learning with the simulation environment to generate several sequence samples to be evaluated. Subsequently, human experts will score and evaluate the sequence samples. Based on the expert scoring results, the algorithm will train the corresponding reward function model, and perform the final anthropomorphic strategy optimization based on the reward model.

[0004] However, although this method based on active scoring by human experts can effectively improve the anthropomorphic performance of the model learning model, from the perspective of practical application, there are still many difficulties: (1) the data quality is unstable and greatly affected by the annotation experts; (2) the overall optimization process is complicated; (3) the knowledge density of the annotated data is low and the training algorithm is inefficient. Based on the above difficulties, the stability of the anthropomorphic effect is poor, and the overall efficiency of the anthropomorphic effect is low. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a method, device, electronic device and storage medium for controlling the behavior of a virtual character, which can significantly improve the stability of the anthropomorphic effect and significantly improve the overall efficiency of the anthropomorphic effect.

[0006] In a first aspect, an embodiment of the present invention provides a behavior control method for a virtual character, comprising:

[0007] Obtaining a visualized game image of a current game environment and quantifiable game features associated with a virtual character in the game environment;

[0008] Determining the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through the target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on the large language model;

[0009] The virtual character is controlled to execute the target game behavior in the game environment so that the virtual character interacts with the game environment.

[0010] In a second aspect, an embodiment of the present invention further provides a behavior control device for a virtual character, comprising:

[0011] A data acquisition module, used to acquire a visual game image of a current game environment and quantifiable game features associated with a virtual character in the game environment;

[0012] A behavior determination module, used to determine the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through a target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on a large language model;

[0013] The interaction module is used to control the virtual character to perform the target game behavior in the game environment so that the virtual character interacts with the game environment.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0016] The embodiment of the present invention provides a method, device, electronic device and storage medium for controlling the behavior of a virtual character. First, a visualized game image of the game environment at the current moment and quantifiable game features associated with the virtual character in the game environment are obtained. Then, the target strategy behavior model obtained through training based on a large language model is used to determine the target game behavior to be executed by the virtual character according to the visualized game image and the quantifiable game features, and then the virtual character is controlled to execute the target game behavior in the game environment so that the virtual character interacts with the game environment. The above method uses a large language model to train a strategy behavior model, which can effectively improve the problems of unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm in the training process. The trained target strategy behavior model is applied to the anthropomorphism process to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a method for controlling the behavior of a virtual character provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a large language model pre-training process provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structure of a LoRA model provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a preference prompt paradigm and a suggestion prompt paradigm provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the structure of a virtual character behavior control device provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] At present, although the method based on active scoring by human experts can effectively improve the humanistic performance of the model learning model, there are still many difficulties from the perspective of practical application:

[0028] (1) Data quality is unstable and greatly affected by annotation experts: Different from general image annotation, anthropomorphic style annotation requires experts to have a deeper understanding of the anthropomorphic behavior of game NPCs, understand what human players' performance is like, and what kind of NPC performance is like a human. This puts a high constraint on the ability of annotation experts. In addition, anthropomorphism itself is a subjective evaluation. Different experts' anthropomorphic evaluations of different pedestrians are prone to deviations. How to unify experts' anthropomorphic preferences is still a difficult problem to solve.

[0029] (2) The overall optimization process is complicated: Since the reinforcement learning method based on human feedback is an optimization method based on human-in-the-loop, the entire method is heavily dependent on the expert's labeling efficiency. In order to ensure high-quality anthropomorphic optimization results, a large amount of data is usually required to be labeled, making the overall labeling and quality inspection process too complicated and inefficient.

[0030] (3) Low knowledge density of labeled data and low efficiency of training algorithms: Existing methods only optimize game behavior through preference labeling results, that is, experts compare two NPC behavior segments and select the relatively human-like behavior segment. Compared with selecting more human-like segments, providing human suggestions on how to operate more like a human is more dense, but this method places too high demands on labeling experts and is often difficult to implement. Game behavior optimization based on preference labeling results is difficult to achieve efficient training efficiency due to the low knowledge density of training data.

[0031] Based on this, the present invention provides a method, device, electronic device and storage medium for controlling the behavior of a virtual character, which can significantly improve the stability of the anthropomorphic effect and significantly improve the overall efficiency of the anthropomorphic effect.

[0032] To facilitate understanding of this embodiment, a behavior control method for a virtual character disclosed in an embodiment of the present invention is first described in detail. Figure 1 The flowchart of a method for controlling the behavior of a virtual character is shown in FIG. 1 , and the method mainly includes the following steps S102 to S106:

[0033] Step S102, obtaining a visualized game image of the current game environment and quantifiable game features associated with a virtual character in the game environment.

[0034] Among them, the visualized game image is the game screen provided to the user through the terminal's graphical user interface, which may specifically include the game scene and small map visible to the player; the virtual character may be an NPC (non-player character); the quantifiable game feature is the game data carrying specific numerical values, including the NPC's attribute characteristics and related environmental information, the attribute information includes the NPC's health, position, speed, attack power, buff, etc., and the environmental information includes time, map information, the position of other NPCs, speed information, etc.

[0035] Step S104, determining the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through the pre-trained target strategy behavior model, wherein the target strategy behavior model is trained based on the large language model.

[0036] In one embodiment, the above-mentioned visualized game image and quantifiable game features are input into the target strategy behavior model to obtain the target game behavior output by the target strategy behavior model.

[0037] In practical applications, the initial strategy behavior model can be trained in advance to obtain the desired target strategy behavior model. During the training process, the output of the initial strategy behavior model can be guided by the pre-trained target large language model and the pre-built anthropomorphic prompt paradigm, so as to train the initial strategy behavior model and obtain the target strategy behavior model.

[0038] Step S106, controlling the virtual character to execute the target game behavior in the game environment, so that the virtual character interacts with the game environment.

[0039] The target game behavior is the simulated game behavior of a real player, which can also be called anthropomorphic behavior, such as advancing (moving forward), retreating, attacking, defending, interacting with other virtual objects, etc. For example, assuming that the target game behavior is moving forward, the virtual character will be controlled to move forward in the game environment.

[0040] The behavior control method of a virtual character provided by an embodiment of the present invention utilizes a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm. The target strategy behavior model obtained through training is applied to the anthropomorphism process, so as to utilize the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0041] The embodiment of the present invention mainly solves how to efficiently and subjectively annotate the game behavior of NPCs, and thereby optimize the training efficiency of the algorithm. Specifically, the embodiment of the present invention solves the following problems: (1) Subjective anthropomorphic annotation not based on human-in-the-loop: Anthropomorphic data annotation based on human-in-the-loop has problems such as unstable data quality, complicated data acquisition and quality inspection processes, etc. Anthropomorphic data annotation is a subjective style annotation, and traditional scripting schemes are difficult to achieve competent recognition of anthropomorphic style. Therefore, there is a need for an anthropomorphic annotation that is not based on subjective anthropomorphic annotation of human-in-the-loop to significantly improve the efficiency and effect of subjective annotation of anthropomorphic style. (2) Anthropomorphic optimization algorithm based on non-expert annotation: Traditional preference-based anthropomorphic style optimization schemes only optimize game behaviors through preference data of two NPC behavior trajectories, which results in low overall training efficiency. To this end, it is very necessary to explore a more efficient anthropomorphic training scheme.

[0042] In view of the above problems, an embodiment of the present invention proposes a method for training a target strategy behavior model, which mainly includes: (1) a language model Finetune that combines video segment encoding based on ViT Embedding with game environment semantic information; (2) NPC anthropomorphic prompt construction; (3) NPC game behavior optimization based on large language model response.

[0043] For ease of understanding, the present invention provides an implementation method for training a target strategy behavior model, specifically referring to the following steps 1 to 4:

[0044] Step 1: pre-train and adjust the large language model based on the first training data set to obtain a target large language model. The first training data set includes visualized game images and quantifiable game features generated by players during the game, that is, real human player data.

[0045] The embodiment of the present invention proposes a language model Finetune that combines video segment encoding based on ViT Embedding with semantic information of the game environment, with the aim of optimizing the large language model's understanding of the game so that it can more accurately understand the game behavior of NPCs in the game. Different from the traditional model fine-tuning scheme that is performed solely on the large language model itself, the embodiment of the present invention draws on the ideas of PaLM-E in the field of robot control and introduces Vision-Transformer (ViT) to extract game features from actual game screens, so as to enhance the large language model's ability to understand the game. Specifically, model pre-training includes pre-training data set construction, LLM model optimization combining ViT and Low-Rank, and model Finetune training.

[0046] For ease of understanding, the embodiment of the present invention provides an implementation method for pre-training and adjusting a large language model based on a first training data set to obtain a target large language model, see steps 1.1 to 1.3 below:

[0047] Step 1.1, pre-train the large language model based on the first training data set to obtain a pre-trained large language model. In the specific implementation, in order to improve the model's understanding of human behavior in the game, this embodiment uses real human player data to construct a pre-training data set. This embodiment records the game screen of real game players in the game, game state features that can be vectorized, text features processed by the scene description unit and the task description unit, and the actual actions performed by the players. When constructing the large language model output, compared to directly using human actions to construct it, this embodiment maps the player's actions into text instructions. For example, the action performed by the player is to attack the enemy in the right front. This embodiment maps the action to a text feature of "facing the current state, selecting the right front as the execution object, and selecting work as the execution action is the most appropriate."

[0048] Typical large language models can only process text features. Therefore, the present invention introduces Vision-Transformer (ViT) to process game image features. Figure 2 The schematic diagram of a large language model pre-training process shown includes the following (1) to (2):

[0049] (1) Performing ViT feature encoding on the visualized game images in the first training data set to obtain image encoding features; and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features; and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features.

[0050] In practical applications, we first obtain the current visual game image (i.e., image features) and related quantifiable game features from the game environment, such as environment status, NPC status, etc. Then, we vector encode the visual game image and quantifiable game features, specifically: we perform feature encoding based on the ViT network on the visual game image to divide the complete game image into several game sub-graphs (i.e., image encoding features); we perform scene description encoding and task description encoding on the quantifiable game features, thus obtaining scene description encoding features and task description encoding features.

[0051] For example, please see Figure 2 , Figure 2 Examples of scene description coding features are shown, including "You see 2 opponents", "Opposite 1 is 2 meters in front of you on the right", "Opposite 2 is 3 meters in front of you on the left", "Opposite is holding a Fang Tian Hua Ji", etc. Figure 2 Examples of task description encoding features are also illustrated, including "Situation 1: You previously chose to attack the person in front of you on the left. If you were a human player, what would you consider doing?" and "Situation 2: Faced with this situation, you did not take any task action before. What would you do now?"

[0052] (2) Pre-training a large language model based on image coding features, scene description coding features, and task description coding features to obtain a pre-trained large language model.

[0053] In practical applications, the game subgraph is encoded through a neural network vector and input into a large language model to assist in understanding the game state. At the same time, the scene description encoding features and task description encoding features are input into the large language model. The large language model responds by outputting behavior description text and converts the behavior description text into executable NPC actions, followed by NPC environment interaction.

[0054] For example, please see Figure 2 , Figure 2 It shows that the behavior description text output by the large language model may be "Faced with situation 1, I will turn to attack the opponent who is relatively close to the right front", "Faced with situation 2, I will directly attack the enemy in the right front because he is closer", etc.

[0055] Furthermore, the large language model is pre-trained based on its output and the actual behaviors of real human players.

[0056] Step 1.2, fix the weight of the pre-trained large language model, and tune the pre-trained large language model through the dimension reduction matrix and the dimension increase matrix to obtain the tuned large language model.

[0057] Optionally, Low-Rank (LoRA) is a classic pre-trained model optimization solution, see Figure 3 The structural diagram of a LoRA model shown in the figure shows that compared with the traditional model Finetune scheme, which optimizes the parameters of the pre-trained model itself, LoRA fixes the weights of the pre-trained model and achieves model tuning by optimizing the dimension reduction matrix A and the dimension increase matrix B. This optimization scheme can effectively reduce the parameters of the pre-trained model while ensuring the training effect.

[0058] Step 1.3: adjust the tuned large language model through the autoregressive algorithm to obtain the target large language model.

[0059] Optionally, during the model Finetune training phase, the embodiment of the present invention uses a classic autoregressive approach to perform model Finetune optimization.

[0060] Step 2: Determine multiple behavior segments to be evaluated based on the second training data set through the initial strategy behavior model, wherein the second training data set includes a visualized game image of the game environment in an offline state and quantifiable game features associated with the virtual characters in the game environment.

[0061] In one embodiment, the visualized game image of the offline game environment and the quantifiable game features associated with the NPC are input into the initial strategy behavior model, and the initial strategy behavior model will output multiple game behaviors, that is, multiple behavior segments to be evaluated.

[0062] Step 3: Determine the output result through the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated. The anthropomorphic prompt paradigm includes the preference prompt paradigm and the suggestion prompt paradigm, and the output result includes the behavior preference result and the behavior suggestion result.

[0063] After completing Feintune of the large language model, the solution then proceeded to construct the NPC anthropomorphic prompt in order to allow the model to output more appropriate game behavior suggestions. In order to fully utilize the understanding ability of the large language model and broaden the knowledge density of the labeled data, the NPC anthropomorphic prompt construction phase will construct the preference prompt paradigm and the suggestion prompt paradigm respectively according to the characteristics of the anthropomorphic decision-making behavior generation problem.

[0064] See also Figure 4 The diagrams of a preference prompt paradigm and a suggestion prompt paradigm are shown in the figure. Figure 4 An example of a preference prompt paradigm and a suggestion prompt paradigm are provided respectively.

[0065] In one example, the preference prompt pattern is described as follows:

[0066] "You are now an expert in evaluating the anthropomorphism of game NPCs. Please evaluate the following two clips and choose the most human-like decision-making behavior;

[0067] Clip A: 1. You see two opponents; 2. Opponent 1 is 2 meters in front of you on the right; 3. Opponent 2 is 3 meters in front of you on the left; 4. The opponent is holding a Fang Tian Hua Ji;

[0068] The NPC's decision object is opponent 1, and the decision behavior is attack;

[0069] Clip A: 1. You see two opponents; 2. Opponent 1 is 2 meters in front of you on the right; 3. Opponent 2 is 3 meters in front of you on the left; 4. The opponent is holding a Fang Tian Hua Ji;

[0070] The NPC's decision object is opponent 2, and the decision behavior is attack;

[0071] As an expert in anthropomorphic assessment, what segment do you think is most appropriate?

[0072] In one example, the suggestion prompt pattern is described as follows:

[0073] “You are now an expert in game NPC anthropomorphism assessment. Please give us some suggestions on the following NPC decision-making snippet;

[0074] 1. You see 2 opponents; 2. Opponent 1 is 2 meters in front of you on the right; 3. Opponent 2 is 3 meters in front of you on the left; 4. The opponent is holding a Fang Tian Hua Ji;

[0075] As an anthropomorphic assessment expert, what do you think are the most appropriate decision-making objects and decision-making behaviors for the NPC at this time?

[0076] exist Figure 4 Based on this, the embodiments of the present invention respectively provide implementation methods for the target large language model to determine the output result based on the preference prompt paradigm and the suggestion prompt paradigm, see the following methods 1 to 2:

[0077] Method 1, based on the preference prompt paradigm: The preference prompt paradigm is a preference judgment scheme, whose input is two behavior segments to be judged, and the output is the judgment result of human experts or models. Specifically, the behavior pair to be evaluated can be determined from multiple behavior segments to be evaluated, and then the preference prompt paradigm and the behavior pair to be evaluated can be input into the target large language model, so that the target large language model outputs the behavior preference result.

[0078] In the embodiment of the present invention, the initial strategy behavior model pre-trained by reinforcement learning or imitation learning interacts with the game environment to generate multiple behavior segments to be evaluated. Subsequently, similar behavior segments are selected through artificial rules to construct behavior pairs to be evaluated. Finally, the behavior pairs to be evaluated and the preference prompt paradigm are input into the large language model to obtain the behavior preference results.

[0079] Method 2, based on the suggestion prompt paradigm: The suggestion prompt expects the large language model to directly make reasonable behavior suggestions based on the current behavior of the NPC. Specifically, the behavior pair to be evaluated can be determined from multiple segments of behavior to be evaluated; through the target large language model, the behavior suggestion result matching the suggestion prompt paradigm is output based on the behavior pair to be evaluated. For example, the behavior pair to be evaluated is input into the target large language model so that the target large prediction model outputs the behavior suggestion result matching the suggestion prompt paradigm.

[0080] Step 4: Train the initial strategy behavior model based on the output results to obtain the target strategy behavior model. In one embodiment, after obtaining the behavior preference results and behavior suggestion results of the language model, the NPC game behavior optimization based on the large language model response will apply these results to the game behavior optimization of the NPC strategy behavior model.

[0081] In one example, based on the behavior preference results output by the large language model, the initial strategy behavior model can be trained according to the following steps a to b:

[0082] In step a, a comparative loss value is determined based on the behavior pair to be evaluated and the behavior preference result through a comparative loss function, and the network parameters of the initial strategy behavior model are adjusted based on the comparative loss value.

[0083] In the specific implementation, the reinforcement learning method based on human feedback can be used to optimize the game behavior, which includes two stages: reward model training and game behavior optimization based on reward function. In the reward model training, the solution builds the model by building a value function branch on the large language model of Finetune, and optimizes the reward model by comparing the loss function. The comparison loss function is as follows:

[0084]

[0085] Where x and y represent the NPC behavior fragment and behavior preference result, respectively. θ represents the value of the reward model with parameter θ, and σ represents the sigmoid activation function.

[0086] Step b: using the proximal strategy optimization algorithm, anthropomorphic strategy optimization is performed on the adjusted initial strategy behavior model to obtain the target strategy behavior model.

[0087] In a specific implementation, after completing the reward function training, the embodiment of the present invention uses a reinforcement learning algorithm based on near-segment strategy optimization to perform anthropomorphic strategy optimization to achieve game behavior improvement.

[0088] In one example, based on the behavior suggestion results output by the large language model, the initial strategy behavior model can be trained according to the following steps: using an imitation learning algorithm, the initial strategy behavior model is trained based on the behavior suggestion results to obtain a target strategy behavior model. In specific implementation, the embodiment of the present invention uses imitation learning to improve the game behavior of the model.

[0089] In summary, the virtual character behavior control method provided by the embodiment of the present invention has at least the following characteristics:

[0090] (1) Compared with manual crowdsourcing annotation, Finetune, a language model based on the combination of video segment encoding and game environment semantic information based on ViT Embedding, can achieve a comprehensive understanding of anthropomorphic task goals and a comprehensive grasp of human player behavior tendencies based on the understanding ability of the large language model and human preferences. Annotating and correcting NPC behavior through a language model can effectively avoid problems such as unstable data quality and complicated data acquisition and quality inspection processes.

[0091] (2) In order to fully utilize the understanding ability of the large language model and broaden the knowledge density of the annotated data, the NPC anthropomorphic prompt construction stage will construct preference prompts and suggestion prompts based on the characteristics of the anthropomorphic decision-making behavior generation problem. The construction of multiple anthropomorphic prompts can provide a data basis for the subsequent improvement of anthropomorphic training efficiency.

[0092] (3) Based on preference prompts and suggestion prompts, the NPC game behavior optimization method based on large language model response can significantly improve the optimization efficiency of game behavior and the final anthropomorphic effect through a joint optimization method combining reinforcement learning and imitation learning.

[0093] Based on the above embodiments, the present invention provides a device for controlling the behavior of a virtual character. Figure 5 The structure diagram of a virtual character behavior control device shown in FIG. 1 mainly includes the following parts:

[0094] A data acquisition module 502 is used to acquire a visualized game image of a current game environment and quantifiable game features associated with a virtual character in the game environment;

[0095] The behavior determination module 504 is used to determine the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features by using the target strategy behavior model obtained by pre-training; wherein the target strategy behavior model is obtained by training based on the large language model;

[0096] The interaction module 506 is used to control the virtual character to perform a target game behavior in the game environment so that the virtual character interacts with the game environment.

[0097] The behavior control device for a virtual character provided by an embodiment of the present invention utilizes a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm. The target strategy behavior model obtained through training is applied to the anthropomorphism process, so as to utilize the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0098] In one embodiment, a model training module is further included, which is used to:

[0099] Pre-training and adjusting the large language model based on the first training data set to obtain a target large language model; wherein the first training data set includes visualized game images and quantifiable game features generated by the player during the game process (i.e., in an online game state);

[0100] Determining a plurality of behavior segments to be evaluated based on a second training data set through an initial strategy behavior model; wherein the second training data set includes a visualized game image of an offline game environment and quantifiable game features associated with a virtual character in the game environment;

[0101] The target large language model is used to determine the output results based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated.

[0102] The initial strategy behavior model is trained based on the output results to obtain the target strategy behavior model.

[0103] In one embodiment, the model training module is further used to:

[0104] Pre-training the large language model based on the first training data set to obtain a pre-trained large language model;

[0105] The weight of the pre-trained large language model is fixed, and the pre-trained large language model is optimized by using a dimensionality reduction matrix and a dimensionality increase matrix to obtain an optimized large language model;

[0106] The tuned large language model is adjusted through the autoregressive algorithm to obtain the target large language model.

[0107] In one embodiment, the model training module is further used to:

[0108] Performing ViT feature encoding on the visualized game image in the first training data set to obtain image encoding features;

[0109] and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features;

[0110] and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features;

[0111] The large language model is pre-trained based on the image coding features, the scene description coding features and the task description coding features to obtain a pre-trained large language model.

[0112] In one embodiment, the anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the model training module is also used for:

[0113] Determine a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0114] The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

[0115] In one embodiment, the model training module is further used to:

[0116] By using the contrast loss function, a contrast loss value is determined based on the behavior pair to be evaluated and the behavior preference result;

[0117] Adjust the network parameters of the initial strategy behavior model based on the contrast loss value;

[0118] The proximal policy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0119] In one embodiment, the anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the model training module is also used for:

[0120] Determine a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0121] Through the target large language model, the behavior suggestion results matching the Prompt paradigm are output based on the behavior to be evaluated;

[0122] In one embodiment, the model training module is further used to:

[0123] Using the imitation learning algorithm, the initial strategy behavior model is trained based on the behavior suggestion results to obtain the target strategy behavior model.

[0124] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0125] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program. When the computer program is run by the processor, it executes:

[0126] A behavior control method for a virtual character, comprising:

[0127] Obtaining a visual game image of the current game environment and quantifiable game features associated with a virtual character in the game environment;

[0128] Determining the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through the target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on the large language model;

[0129] The virtual character is controlled to execute target game behaviors in the game environment so that the virtual character interacts with the game environment.

[0130] In one embodiment, the step of training the target strategy behavior model includes:

[0131] Pre-training and adjusting the large language model based on the first training data set to obtain a target large language model; wherein the first training data set includes visualized game images and quantifiable game features generated during the player's game process;

[0132] Determining a plurality of behavior segments to be evaluated based on a second training data set through an initial strategy behavior model; wherein the second training data set includes a visualized game image of an offline game environment and quantifiable game features associated with a virtual character in the game environment;

[0133] The target large language model is used to determine the output results based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated.

[0134] The initial strategy behavior model is trained based on the output results to obtain the target strategy behavior model.

[0135] In one embodiment, the step of pre-training and adjusting the large language model based on the first training data set to obtain a target large language model includes:

[0136] Pre-training the large language model based on the first training data set to obtain a pre-trained large language model;

[0137] The weight of the pre-trained large language model is fixed, and the pre-trained large language model is optimized by using a dimensionality reduction matrix and a dimensionality increase matrix to obtain an optimized large language model;

[0138] The tuned large language model is adjusted through the autoregressive algorithm to obtain the target large language model.

[0139] In one embodiment, the step of pre-training the large language model based on the first training data set to obtain the pre-trained large language model includes:

[0140] Performing ViT feature encoding on the visualized game image in the first training data set to obtain image encoding features;

[0141] and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features;

[0142] and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features;

[0143] The large language model is pre-trained based on the image coding features, the scene description coding features and the task description coding features to obtain a pre-trained large language model.

[0144] In one embodiment, the anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, including:

[0145] Determine a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0146] The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

[0147] In one embodiment, the initial strategy behavior model is trained based on the output result to obtain the target strategy behavior model, including:

[0148] By using the contrast loss function, a contrast loss value is determined based on the behavior pair to be evaluated and the behavior preference result;

[0149] Adjust the network parameters of the initial strategy behavior model based on the contrast loss value;

[0150] The proximal policy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0151] In one embodiment, the anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, and further includes:

[0152] Determine a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0153] Through the target large language model, the behavior suggestion results matching the Prompt paradigm are output based on the behavior to be evaluated;

[0154] In one embodiment, the initial strategy behavior model is trained based on the output result to obtain the target strategy behavior model, further comprising:

[0155] Using the imitation learning algorithm, the initial strategy behavior model is trained based on the behavior suggestion results to obtain the target strategy behavior model.

[0156] The electronic device provided by the embodiment of the present invention uses a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm. The trained target strategy behavior model is applied to the anthropomorphism process, so as to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0157] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0158] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0159] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0160] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0161] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0162] The computer program product of a readable storage medium provided in an embodiment of the present invention includes a computer readable storage medium storing program code, and the instructions included in the program code can be used to execute:

[0163] A behavior control method for a virtual character, comprising:

[0164] Obtaining a visual game image of the current game environment and quantifiable game features associated with a virtual character in the game environment;

[0165] Determining the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through the target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on the large language model;

[0166] The virtual character is controlled to execute target game behaviors in the game environment so that the virtual character interacts with the game environment.

[0167] In one embodiment, the step of training the target strategy behavior model includes:

[0168] Pre-training and adjusting the large language model based on the first training data set to obtain a target large language model; wherein the first training data set includes visualized game images and quantifiable game features generated during the player's game process;

[0169] Determining a plurality of behavior segments to be evaluated based on a second training data set through an initial strategy behavior model; wherein the second training data set includes a visualized game image of an offline game environment and quantifiable game features associated with a virtual character in the game environment;

[0170] The target large language model is used to determine the output results based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated.

[0171] The initial strategy behavior model is trained based on the output results to obtain the target strategy behavior model.

[0172] In one embodiment, the step of pre-training and adjusting the large language model based on the first training data set to obtain a target large language model includes:

[0173] Pre-training the large language model based on the first training data set to obtain a pre-trained large language model;

[0174] The weight of the pre-trained large language model is fixed, and the pre-trained large language model is optimized by using a dimensionality reduction matrix and a dimensionality increase matrix to obtain an optimized large language model;

[0175] The tuned large language model is adjusted through the autoregressive algorithm to obtain the target large language model.

[0176] In one embodiment, the step of pre-training the large language model based on the first training data set to obtain the pre-trained large language model includes:

[0177] Performing ViT feature encoding on the visualized game image in the first training data set to obtain image encoding features;

[0178] and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features;

[0179] and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features;

[0180] The large language model is pre-trained based on the image coding features, the scene description coding features and the task description coding features to obtain a pre-trained large language model.

[0181] In one embodiment, the anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, including:

[0182] Determine a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0183] The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

[0184] In one embodiment, the initial strategy behavior model is trained based on the output result to obtain the target strategy behavior model, including:

[0185] By using the contrast loss function, a contrast loss value is determined based on the behavior pair to be evaluated and the behavior preference result;

[0186] Adjust the network parameters of the initial strategy behavior model based on the contrast loss value;

[0187] The proximal policy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0188] In one embodiment, the anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, and further includes:

[0189] Determine a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0190] Through the target large language model, the behavior suggestion results matching the Prompt paradigm are output based on the behavior to be evaluated;

[0191] In one embodiment, the initial strategy behavior model is trained based on the output result to obtain the target strategy behavior model, further comprising:

[0192] Using the imitation learning algorithm, the initial strategy behavior model is trained based on the behavior suggestion results to obtain the target strategy behavior model.

[0193] The readable storage medium provided by the embodiment of the present invention uses a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm. The target strategy behavior model obtained through training is applied to the anthropomorphism process, so as to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0194] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0195] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for controlling the behavior of a virtual character, characterized in that: include: Obtaining a visualized game image of a current game environment and quantifiable game features associated with a virtual character in the game environment; Determining the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through the target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on the large language model; The virtual character is controlled to execute the target game behavior in the game environment so that the virtual character interacts with the game environment.

2. The method for controlling the behavior of a virtual character according to claim 1, characterized in that: The training steps of the target strategy behavior model include: Pre-training and adjusting the large language model based on a first training data set to obtain a target large language model; wherein the first training data set includes visualized game images and quantifiable game features generated during a player's game process; Determine, through the initial strategy behavior model, a plurality of behavior segments to be evaluated based on a second training data set; wherein the second training data set includes a visualized game image of an offline game environment and quantifiable game features associated with the virtual character located in the game environment; Determine the output result through the target large language model based on the pre-constructed anthropomorphic prompt paradigm and the multiple behavior segments to be evaluated; The initial strategy behavior model is trained based on the output result to obtain a target strategy behavior model.

3. The method for controlling the behavior of a virtual character according to claim 2, characterized in that: The step of pre-training and adjusting the large language model based on the first training data set to obtain a target large language model includes: Pre-training the large language model based on the first training data set to obtain a pre-trained large language model; The weight of the pre-trained large language model is fixed, and the pre-trained large language model is optimized by using a dimensionality reduction matrix and a dimensionality increase matrix to obtain an optimized large language model; The tuned large language model is adjusted by an autoregressive algorithm to obtain a target large language model.

4. The method for controlling the behavior of a virtual character according to claim 2, characterized in that: The step of pre-training the large language model based on the first training data set includes: Performing ViT feature encoding on the visualized game image in the first training data set to obtain image encoding features; and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features; and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features; The large language model is pre-trained based on the image coding features, the scene description coding features and the task description coding features to obtain a pre-trained large language model.

5. The method for controlling the behavior of a virtual character according to claim 2, characterized in that: The anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the output result is determined by the target large language model based on the pre-constructed anthropomorphic prompt paradigm and the multiple behavior segments to be evaluated, including: Determining a pair of behaviors to be evaluated from a plurality of the behaviors to be evaluated; The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model, so that the target large language model outputs a behavior preference result.

6. The method for controlling the behavior of a virtual character according to claim 5, characterized in that: The initial strategy behavior model is trained based on the output result to obtain a target strategy behavior model, including: Determining a contrast loss value based on the behavior pair to be evaluated and the behavior preference result by using a contrast loss function; Adjusting the network parameters of the initial strategy behavior model based on the contrast loss value; The proximal strategy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial strategy behavior model to obtain a target strategy behavior model.

7. The method for controlling the behavior of a virtual character according to claim 2, characterized in that: The anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the output result is determined by the target large language model based on the pre-constructed anthropomorphic prompt paradigm and the multiple behavior segments to be evaluated, and further includes: Determining a pair of behaviors to be evaluated from a plurality of the behaviors to be evaluated; Outputting, through the target large language model, a behavior suggestion result matching the suggestion-style Prompt paradigm based on the behavior to be evaluated; The initial strategy behavior model is trained based on the output result to obtain a target strategy behavior model, further comprising: The initial strategy behavior model is trained based on the behavior suggestion results using an imitation learning algorithm to obtain a target strategy behavior model.

8. A virtual character behavior control device, characterized in that: include: A data acquisition module, used to acquire a visual game image of a current game environment and quantifiable game features associated with a virtual character in the game environment; A behavior determination module, used to determine the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through a target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on a large language model; The interaction module is used to control the virtual character to perform the target game behavior in the game environment so that the virtual character interacts with the game environment.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images