Method and apparatus for controlling behavior of virtual character, and electronic device and storage medium

Abstract

A method and apparatus for controlling a behavior of a virtual character, and an electronic device and a storage medium, which relate to the technical field of games. The method comprises: acquiring a visual game image of a game environment at the current moment and a quantifiable game feature associated with a virtual character which is located in the game environment; using a target strategy behavior model obtained by means of pre-training, determining a target game behavior to be executed of the virtual character on the basis of the visual game image and the quantifiable game feature, wherein the target strategy behavior model is obtained by means of performing training on the basis of a large language model; and controlling the virtual character to execute the target game behavior in the game environment, such that the virtual character interacts with the game environment. By means of the present disclosure, the stability of the anthropomorphic effect can be significantly improved, and the overall anthropomorphic efficiency can be significantly improved.

Description

Virtual character behavior control method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202311338246.5, filed on October 16, 2023, entitled “Virtual Character Behavior Control Method, Device, Electronic Device and Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of game technology, and in particular to a method, device, electronic device, and storage medium for controlling the behavior of a virtual character. Background Art

[0004] To achieve anthropomorphic strategy generation, the reward function of the reinforcement learning algorithm often requires careful design. Although carefully tuning the reward function can address the problem of some agents' behavior not being sufficiently similar to that of humans, the inherent subjectivity of anthropomorphism makes it difficult to design a specific reward function to optimize agent behavior. Therefore, researchers often use imitation learning methods that are not based on reward functions to generate anthropomorphic agent behavior. Imitation learning, as an offline data optimization scheme, only requires the expert's historical behavior sequence to imitate the expert's strategy, thus achieving objective imitation of the expert's behavior.

[0005] Although imitation learning can learn game behaviors from an objective perspective, the complexity and diversity of human data make it difficult for imitation learning to effectively imitate more detailed game behaviors. For this reason, most related technologies will perform secondary optimization on the strategies learned by imitation learning from a subjective perspective. Reinforcement learning based on human feedback is a reinforcement learning algorithm that uses human subjective preferences to optimize the model's strategy. Specifically, it first interacts the strategy learned by imitation learning with the simulation environment to generate several sequence samples to be evaluated. Subsequently, human experts will score and evaluate the sequence samples. Based on the expert scoring results, the algorithm will train the corresponding reward function model and perform the final anthropomorphic strategy optimization based on this reward model.

[0006] However, although this method based on active scoring by human experts can effectively improve the anthropomorphic performance of the model learning model, it still has many difficulties from the perspective of practical application: (1) Data quality is unstable and greatly affected by the annotation experts; (2) The overall optimization process is complicated; (3) The knowledge density of the labeled data is low and the training algorithm is inefficient. Due to the above difficulties, the anthropomorphic effect is less stable and the overall efficiency of anthropomorphism is low.

[0007] Summary of the Invention

[0008] In view of this, the purpose of the present disclosure is to provide a method, device, electronic device and storage medium for controlling the behavior of a virtual character, which can significantly improve the stability of the anthropomorphic effect and significantly improve the overall efficiency of the anthropomorphic effect.

[0009] In a first aspect, an embodiment of the present disclosure provides a method for controlling the behavior of a virtual character, comprising:

[0010] Obtaining a visual game image of a current game environment and quantifiable game features associated with a virtual character within the game environment;

[0011] Determining a target game behavior to be executed by the virtual character based on the visualized game image and the quantifiable game features using a pre-trained target strategy behavior model; wherein the target strategy behavior model is trained based on a large language model;

[0012] The virtual character is controlled to perform the target game behavior in the game environment, so that the virtual character interacts with the game environment.

[0013] In a second aspect, the present disclosure also provides a device for controlling the behavior of a virtual character, including:

[0014] A data acquisition module is configured to acquire a visual game image of a current game environment and quantifiable game features associated with a virtual character within the game environment;

[0015] a behavior determination module configured to execute a pre-trained target strategy behavior model to determine a target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features; wherein the target strategy behavior model is trained based on a large language model;

[0016] The interaction module is configured to control the virtual character to perform the target game behavior in the game environment, so that the virtual character interacts with the game environment.

[0017] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0019] The disclosed embodiments provide a method, device, electronic device, and storage medium for controlling the behavior of a virtual character. The method first obtains a visual game image of the game environment at the current moment and quantifiable game features associated with the virtual character in the game environment. Then, through a target strategy behavior model obtained through training based on a large language model, the target game behavior to be executed by the virtual character is determined according to the visual game image and the quantifiable game features. The method then controls the virtual character to execute the target game behavior in the game environment so that the virtual character interacts with the game environment. The above method uses a large language model to train a strategy behavior model, which can effectively improve the problems of unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm in the training process. The trained target strategy behavior model is applied to the anthropomorphism process to use the target strategy behavior model to determine the required target game behavior based on the visual game image and quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0020] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or understood by practicing the present disclosure. The objectives and other advantages of the present disclosure are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0021] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the following briefly introduces the drawings required for use in the specific embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] FIG1 is a flow chart of a method for controlling the behavior of a virtual character according to one embodiment of the present disclosure;

[0024] FIG2 is a schematic diagram of a large language model pre-training process provided by one embodiment of the present disclosure;

[0025] FIG3 is a schematic structural diagram of a LoRA model provided by one embodiment of the present disclosure;

[0026] FIG4 is a schematic diagram of a preference prompt paradigm and a suggestion prompt paradigm provided by one embodiment of the present disclosure;

[0027] FIG5 is a schematic structural diagram of a device for controlling the behavior of a virtual character provided by one embodiment of the present disclosure;

[0028] FIG6 is a schematic structural diagram of an electronic device provided by one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort shall fall within the scope of protection of the present disclosure.

[0030] Currently, although the method based on active scoring by human experts can effectively improve the human-like performance of model learning, it still faces many difficulties from the perspective of practical application:

[0031] (1) Data quality is unstable and greatly influenced by annotation experts: Different from general image annotation, anthropomorphic style annotation requires experts to have a deep understanding of the anthropomorphic behavior of game NPCs, understand what human players behave like, and what kind of NPC behavior is human-like. This places high constraints on the ability of annotation experts. In addition, anthropomorphism itself is a subjective evaluation. Different experts' anthropomorphic evaluations of different pedestrians are prone to deviations. How to unify experts' anthropomorphic preferences remains a difficult problem to solve.

[0032] (2) The overall optimization process is complex: Since the reinforcement learning method based on human feedback is a human-in-the-loop optimization method, the entire method is heavily dependent on the efficiency of expert labeling. To ensure high-quality anthropomorphic optimization results, a large amount of data is usually required to be labeled, making the overall labeling and quality inspection process too complex and inefficient.

[0033] (3) The knowledge density of labeled data is low, and the training algorithm is inefficient: Related methods only optimize game behavior based on preference labeling results. That is, experts compare two NPC behavior segments and select the one that is relatively human-like. Compared with selecting the more human-like segment, providing human-like suggestions on how to operate more like a human is more dense, but this method places too high demands on labeling experts and is often difficult to implement. Game behavior optimization based on preference labeling results is difficult to achieve efficient training efficiency due to the low knowledge density of training data.

[0034] Based on this, the present disclosure implements a method, device, electronic device and storage medium for controlling the behavior of a virtual character, which can significantly improve the stability of the anthropomorphic effect and significantly improve the overall efficiency of the anthropomorphic effect.

[0035] To facilitate understanding of this embodiment, a method for controlling the behavior of a virtual character disclosed in an embodiment of the present disclosure is first described in detail. Referring to a flow chart of a method for controlling the behavior of a virtual character shown in FIG1 , the method mainly includes the following steps S102 to S106:

[0036] Step S102: obtaining a visual game image of the current game environment and quantifiable game features associated with the virtual character in the game environment.

[0037] Among them, the visual game image is the game screen provided to the user through the terminal's graphical user interface, which may specifically include the player's visible game scene and small map, etc.; the virtual character may be an NPC (non-player character); the quantifiable game feature is the game data carrying specific numerical values, including the NPC's attribute characteristics and related environmental information. The attribute information includes the NPC's health, position, speed, attack power, buff, etc., and the environmental information includes time, map information, the position and speed information of other NPCs, etc.

[0038] Step S104: Determine the target game behavior to be executed by the virtual character based on the visualized game image and quantifiable game features using the pre-trained target strategy behavior model, which is trained based on the large language model.

[0039] In one embodiment, the above-mentioned visualized game image and quantifiable game features are input into the target strategy behavior model to obtain the target game behavior output by the target strategy behavior model.

[0040] In practical applications, the initial policy behavior model can be pre-trained to obtain the desired target policy behavior model. During the training process, the pre-trained target large language model and the pre-built anthropomorphic prompt paradigm can be used to guide the output of the initial policy behavior model, thereby training the initial policy behavior model and obtaining the target policy behavior model.

[0041] Step S106: Control the virtual character to perform the target game behavior in the game environment, so that the virtual character interacts with the game environment.

[0042] The target game behavior is the simulated game behavior of a real player, which can also be called anthropomorphic behavior, such as advancing (moving forward), retreating, attacking, defending, interacting with other virtual objects, etc. For example, if the target game behavior is moving forward, the virtual character will be controlled to move forward in the game environment.

[0043] The behavior control method of the virtual character provided by the embodiment of the present disclosure utilizes a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of the training algorithm. The trained target strategy behavior model is applied to the anthropomorphism process, so as to utilize the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0044] The main problem solved by the embodiments of the present disclosure is how to efficiently and subjectively label the game behavior of NPCs and thereby optimize the training efficiency of the algorithm. Specifically, the embodiments of the present disclosure solve the following problems: (1) Subjective anthropomorphic labeling not based on human-in-the-loop: Anthropomorphic data labeling based on human-in-the-loop has problems such as unstable data quality, complicated data acquisition and quality inspection processes. Anthropomorphic data labeling is a subjective style labeling, and traditional scripting schemes are difficult to achieve competent recognition of anthropomorphic styles. Therefore, there is a need for a method that is not based on subjective anthropomorphic labeling of human-in-the-loop to significantly improve the efficiency and effect of subjective anthropomorphic style labeling. (2) Anthropomorphic optimization algorithm based on non-expert labeling: Traditional preference-based anthropomorphic style optimization schemes only optimize game behavior through the preference data of two NPC behavior trajectories, which results in low overall training efficiency. For this reason, it is very necessary to explore a more efficient anthropomorphic training scheme.

[0045] To address the above issues, the present disclosure proposes a method for training a target strategy behavior model, which mainly comprises: (1) a language model Finetune that combines video segment encoding based on ViT Embedding with game environment semantic information; (2) NPC anthropomorphic prompt construction; and (3) NPC game behavior optimization based on large language model response.

[0046] For ease of understanding, the present disclosure provides an implementation method for training a target policy behavior model. For details, see steps 1 to 4 below:

[0047] Step 1: Pre-train and adjust the large language model based on a first training dataset to obtain a target large language model. The first training dataset includes visual game images and quantifiable game features generated by players during gameplay, i.e., real human player data.

[0048] The disclosed embodiment proposes a language model Finetune that combines video segment encoding based on ViT Embedding with semantic information of the game environment. Its purpose is to optimize the large language model's understanding of the game so that it can more accurately understand the game behavior of NPCs in the game. Different from the traditional model fine-tuning scheme that is performed only on the large language model itself, the disclosed embodiment draws on the ideas of PaLM-E in the field of robot control and introduces Vision-Transformer (ViT) to extract game features from actual game screens to enhance the large language model's ability to understand the game. Specifically, model pre-training includes pre-training data set construction, LLM model optimization combining ViT and Low-Rank, and model Finetune training.

[0049] For ease of understanding, the present disclosure provides an implementation method for pre-training and adjusting a large language model based on a first training dataset to obtain a target large language model, as shown in steps 1.1 to 1.3 below:

[0050] Step 1.1, pre-train the large language model based on the first training data set to obtain a pre-trained large language model. In the specific implementation, in order to improve the model's understanding of human behavior in the game, this embodiment uses real human player data to construct a pre-training data set. This embodiment records the game screen of real game players in the game, game state features that can be vectorized, text features processed by the scene description unit and the task description unit, and the actual actions performed by the players. When constructing the large language model output, compared to directly using human actions to construct it, this embodiment maps the player's actions into text instructions. For example, the action performed by the player is to attack the enemy in the right front. This embodiment maps the action to a text feature of "Facing the current state, select the right front as the execution object, and select work as the execution action is the most appropriate."

[0051] Typical large language models can only process text features. Therefore, the present disclosure introduces the Vision-Transformer (ViT) network language to process game image features. Specifically, referring to FIG2 , a schematic diagram of a large language model pre-training process includes the following (1) to (2):

[0052] (1) Performing ViT feature encoding on the visual game images in the first training data set to obtain image encoding features; and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features; and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features.

[0053] In practical applications, the current visual game image (i.e., image features) and related quantifiable game features, such as the environment state and NPC state, are first obtained from the game environment. Vector encoding is then performed on the visual game image and quantifiable game features. Specifically, the visual game image is encoded using a ViT network-based feature encoding to divide the complete game image into several game subgraphs (i.e., image encoding features). The quantifiable game features are then encoded using scene descriptions and task descriptions, resulting in scene description encoding features and task description encoding features.

[0054] For example, please continue to refer to Figure 2, which illustrates examples of scene description coding features, including "you see 2 opponents", "Opponent 1 is 2 meters in front of you on the right", "Opponent 2 is 3 meters in front of you on the left", "Opponent is holding a Fang Tian Hua Ji", etc.; Figure 2 also illustrates examples of task description coding features, including "Situation 1: You previously chose to attack the person in front of you on the left. If you were a human player, what would you consider doing?", "Situation 2: Faced with this situation, you did not take task operations before. What would you do now?", etc.

[0055] (2) Pre-training the large language model based on image coding features, scene description coding features, and task description coding features to obtain a pre-trained large language model.

[0056] In practical applications, the game subgraph is encoded through a neural network vector and input into a large language model to assist in understanding the game state. At the same time, the scene description encoding features and task description encoding features are input into the large language model. The large language model responds by outputting behavior description text and converting the behavior description text into executable NPC actions, followed by NPC environment interaction.

[0057] For example, please continue to refer to Figure 2, which shows that the behavior description text output by the large language model may be "Faced with situation 1, I will turn to attack the opponent who is relatively close to the right front", "Faced with situation 2, I will directly attack the enemy in the right front because he is closer", etc.

[0058] Furthermore, the large language model is pre-trained based on its output and the actual behavior of real human players.

[0059] In step 1.2, the weights of the pre-trained large language model are fixed, and the pre-trained large language model is tuned using a dimensionality reduction matrix and a dimensionality increase matrix to obtain a tuned large language model.

[0060] Alternatively, Low-Rank (LoRA) is a classic pre-trained model optimization scheme. Figure 3 shows a schematic diagram of the LoRA model structure. Compared to the traditional Finetune scheme, which optimizes the parameters of the pre-trained model itself, LoRA fixes the pre-trained model weights and achieves model tuning by optimizing the dimensionality reduction matrix A and the dimensionality increase matrix B. This optimization scheme can effectively reduce the pre-trained model parameters while ensuring training results.

[0061] In step 1.3, the tuned large language model is adjusted using an autoregressive algorithm to obtain the target large language model.

[0062] Optionally, during the model Finetune training phase, the disclosed embodiment uses a classic autoregressive approach to perform model Finetune optimization.

[0063] Step 2: Using the initial strategic behavior model, determine multiple behavior segments to be evaluated based on a second training dataset. The second training dataset includes a visual game image of an offline game environment and quantifiable game features associated with virtual characters within the game environment.

[0064] In one embodiment, the visual game image of the offline game environment and the quantifiable game features associated with the NPC are input into the initial strategy behavior model, and the initial strategy behavior model will output multiple game behaviors, that is, multiple behavior segments to be evaluated.

[0065] Step 3: The target large language model determines the output results based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated. The anthropomorphic prompt paradigm includes the preference prompt paradigm and the suggestion prompt paradigm, and the output results include the behavior preference result and the behavior suggestion result.

[0066] After completing Feintune on the large language model, the solution then proceeded to construct anthropomorphic NPC prompts to enable the model to output more appropriate game behavior suggestions. To fully leverage the large language model's comprehension capabilities and broaden the knowledge density of the annotated data, the NPC anthropomorphic prompt construction phase will construct preference-based prompts and suggestion-based prompts based on the characteristics of the anthropomorphic decision-making behavior generation questions.

[0067] Referring to FIG. 4 , which shows a schematic diagram of a preference prompt paradigm and a suggestion prompt paradigm, FIG. 4 provides examples of a preference prompt paradigm and a suggestion prompt paradigm, respectively.

[0068] In one example, the preference prompt paradigm is described as follows:

[0069] "You are now an expert in evaluating the anthropomorphism of game NPCs. Please evaluate the following two clips and choose the most human-like decision-making behavior;

[0070] Clip A: 1. You see two opponents; 2. Opponent 1 is 2 meters in front of you on the right; 3. Opponent 2 is 3 meters in front of you on the left; 4. The opponent is holding a Fang Tian Hua Ji.

[0071] The NPC's decision object is opponent 1, and the decision action is attack;

[0072] Clip A: 1. You see two opponents; 2. Opponent 1 is 2 meters in front of you on the right; 3. Opponent 2 is 3 meters in front of you on the left; 4. The opponent is holding a Fang Tian Hua Ji.

[0073] The NPC's decision object is opponent 2, and the decision action is attack;

[0074] As an expert in anthropomorphic assessment, what segment do you think is most appropriate?"

[0075] In one example, the suggestion prompt pattern is described as follows:

[0076] "You are now an expert in anthropomorphic evaluation of game NPCs. Please give us your suggestions on the following NPC decision-making snippet.

[0077] 1. You see two opponents; 2. Opponent 1 is 2 meters in front of you on the right; 3. Opponent 2 is 3 meters in front of you on the left; 4. The opponent is holding a Fang Tian Hua Ji;

[0078] As an expert in personified assessment, what do you think are the most appropriate decision-making objects and decision-making behaviors for the NPC at this moment?

[0079] Based on FIG4 , the embodiments of the present disclosure provide implementations for determining output results by the target large language model based on the preference prompt paradigm and the suggestion prompt paradigm, respectively, as shown in the following methods 1 and 2:

[0080] Method 1, based on the preference prompt paradigm: The preference prompt paradigm is a preference judgment scheme that takes two action segments to be judged as input and outputs the judgment result of a human expert or model. Specifically, a pair of actions to be evaluated can be determined from multiple action segments. The preference prompt paradigm and the action pairs to be evaluated are then input into the target large language model, which then outputs the action preference result.

[0081] In this disclosed embodiment, an initial strategic behavior model, pre-trained through reinforcement learning or imitation learning, interacts with the game environment to generate multiple behavior segments to be evaluated. Subsequently, artificial rules are used to select similar behavior segments to construct behavior pairs to be evaluated. Finally, the behavior pairs to be evaluated and the preference prompt paradigm are input into the large language model to obtain the behavior preference results.

[0082] Method 2, based on the suggestion prompt paradigm: The suggestion prompt paradigm expects the large language model to directly provide reasonable behavior suggestions based on the NPC's current behavior. Specifically, a behavior pair to be evaluated can be determined from multiple behavior segments to be evaluated. Based on the behavior pair to be evaluated, the target large language model outputs a behavior suggestion result that matches the suggestion prompt paradigm. For example, the behavior pair to be evaluated is input into the target large language model, so that the target large prediction model outputs a behavior suggestion result that matches the suggestion prompt paradigm.

[0083] Step 4: Train the initial strategy behavior model based on the output results to obtain the target strategy behavior model. In one embodiment, after obtaining the behavior preference results and behavior suggestion results of the language model, the NPC game behavior optimization based on the large language model response will apply these results to the game behavior optimization of the NPC strategy behavior model.

[0084] In one example, based on the behavior preference results output by the large language model, the initial strategy behavior model can be trained according to the following steps a to b:

[0085] In step a, the contrast loss function is used to determine the contrast loss value based on the behavior pair to be evaluated and the behavior preference results, and the network parameters of the initial strategy behavior model are adjusted based on the contrast loss value.

[0086] In practice, a reinforcement learning method based on human feedback can be used to optimize game behavior. This involves two phases: reward model training and reward function-based game behavior optimization. During reward model training, the solution constructs the model by building a value function branch based on the completed Finetune large language model. The reward model is then optimized using a comparative loss function, as shown below:

[0087] Where x and y represent the NPC behavior fragment and behavior preference results respectively, r θ represents the value of the reward model with parameter θ, and σ represents the sigmoid activation function.

[0088] Step b: Using the proximal strategy optimization algorithm, perform anthropomorphic strategy optimization on the adjusted initial strategy behavior model to obtain the target strategy behavior model.

[0089] In specific implementation, after completing the reward function training, the embodiment of the present disclosure uses a reinforcement learning algorithm based on near-segment strategy optimization to perform anthropomorphic strategy optimization to achieve improved gaming behavior.

[0090] In one example, based on the behavior suggestion results output by the large language model, the initial strategy behavior model can be trained according to the following steps: Using an imitation learning algorithm, the initial strategy behavior model is trained based on the behavior suggestion results to obtain a target strategy behavior model. In specific implementation, the embodiments of the present disclosure use imitation learning to improve the model's gaming behavior.

[0091] In summary, the virtual character behavior control method provided by the embodiments of the present disclosure has at least the following characteristics:

[0092] (1) Compared to manual crowdsourcing annotation, Finetune, a language model based on ViT Embedding that combines video segment encoding with game environment semantic information, can achieve a comprehensive understanding of anthropomorphic task goals and fully grasp the behavioral tendencies of human players based on the understanding capabilities of large language models and human preferences. Using language models to annotate and correct NPC behaviors can effectively avoid problems such as unstable data quality and complicated data acquisition and quality inspection processes.

[0093] (2) To fully utilize the understanding capabilities of the large language model and broaden the knowledge density of the annotated data, the NPC anthropomorphic prompt construction phase will construct preference prompts and suggestion prompts based on the characteristics of the anthropomorphic decision-making behavior generation problem. The construction of multiple anthropomorphic prompts can provide a data foundation for subsequent improvement of anthropomorphic training efficiency.

[0094] (3) Based on preference prompts and suggestion prompts, the NPC game behavior optimization method based on large language model response can significantly improve the optimization efficiency of game behavior and the final anthropomorphic effect through a joint optimization method combining reinforcement learning and imitation learning.

[0095] Based on the above embodiments, the present disclosure provides a device for controlling the behavior of a virtual character. Referring to FIG5 , a schematic diagram of the structure of the device for controlling the behavior of a virtual character is shown. The device mainly includes the following parts:

[0096] The data acquisition module 502 is configured to acquire a visual game image of a current game environment and quantifiable game features associated with a virtual character in the game environment;

[0097] The behavior determination module 504 is configured to execute a pre-trained target strategy behavior model to determine a target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features; wherein the target strategy behavior model is trained based on the large language model;

[0098] The interaction module 506 is configured to control the virtual character to perform a target game behavior in the game environment, so that the virtual character interacts with the game environment.

[0099] The behavior control device for a virtual character provided by the embodiment of the present disclosure uses a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of the training algorithm. The target strategy behavior model obtained through training is applied to the anthropomorphism process, so as to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0100] In one embodiment, the present invention further includes a model training module configured to execute:

[0101] Pre-training and adjusting the large language model based on a first training dataset to obtain a target large language model; wherein the first training dataset includes visual game images and quantifiable game features generated by players during a game (i.e., in an online game state);

[0102] Determining, using the initial strategic behavior model, a plurality of behavior segments to be evaluated based on a second training dataset, wherein the second training dataset includes a visual game image of an offline game environment and quantifiable game features associated with a virtual character within the game environment;

[0103] The target language model is used to determine the output based on a pre-built anthropomorphic prompt pattern and multiple behavior segments to be evaluated.

[0104] The initial strategy behavior model is trained based on the output results to obtain the target strategy behavior model.

[0105] In one embodiment, the model training module is further configured to perform:

[0106] Pre-training the large language model based on the first training data set to obtain a pre-trained large language model;

[0107] Fix the weight of the pre-trained large language model and optimize it through dimensionality reduction and dimensionality increase matrices to obtain an optimized large language model.

[0108] The tuned large language model is adjusted through the autoregressive algorithm to obtain the target large language model.

[0109] In one embodiment, the model training module is further configured to perform:

[0110] Performing ViT feature encoding on the visual game image in the first training data set to obtain image encoding features;

[0111] and performing scene description coding on the quantifiable game features in the first training data set to obtain scene description coding features;

[0112] and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features;

[0113] A large language model is pre-trained based on the image coding features, the scene description coding features, and the task description coding features to obtain a pre-trained large language model.

[0114] In one embodiment, the anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the model training module is further configured to execute:

[0115] determining a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0116] The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

[0117] In one embodiment, the model training module is further configured to perform:

[0118] Determine the contrast loss value based on the behavior pair to be evaluated and the behavior preference result through the contrast loss function;

[0119] Adjust the network parameters of the initial strategy behavior model based on the contrast loss value;

[0120] The proximal policy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0121] In one embodiment, the anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the model training module is further configured to execute:

[0122] determining a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0123] Through the target large language model, the behavior suggestion results matching the Prompt paradigm are output based on the behavior to be evaluated;

[0124] In one embodiment, the model training module is further configured to perform:

[0125] Using the imitation learning algorithm, the initial strategy behavior model is trained based on the behavior recommendation results to obtain the target strategy behavior model.

[0126] The device provided in the embodiment of the present disclosure has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding content in the aforementioned method embodiment.

[0127] An embodiment of the present disclosure provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, which, when executed by the processor, executes:

[0128] A behavior control method for a virtual character, comprising:

[0129] Obtaining a visual game image of the current game environment and quantifiable game features associated with a virtual character within the game environment;

[0130] Determining a target game behavior to be executed by the virtual character based on the visualized game image and the quantifiable game features using a pre-trained target strategy behavior model; wherein the target strategy behavior model is trained based on a large language model;

[0131] The virtual character is controlled to perform target game behaviors within the game environment so that the virtual character interacts with the game environment.

[0132] In one embodiment, the step of training the target policy behavior model includes:

[0133] Pre-training and adjusting the large language model based on a first training dataset to obtain a target large language model; wherein the first training dataset includes visual game images and quantifiable game features generated by players during game play;

[0134] Determining, using the initial strategic behavior model, a plurality of behavior segments to be evaluated based on a second training dataset, wherein the second training dataset includes a visual game image of an offline game environment and quantifiable game features associated with a virtual character within the game environment;

[0135] The target language model is used to determine the output based on a pre-built anthropomorphic prompt pattern and multiple behavior segments to be evaluated.

[0136] The initial strategy behavior model is trained based on the output results to obtain the target strategy behavior model.

[0137] In one embodiment, the step of pre-training and adjusting the large language model based on the first training dataset to obtain a target large language model includes:

[0138] Pre-training the large language model based on the first training data set to obtain a pre-trained large language model;

[0139] Fix the weight of the pre-trained large language model and optimize it through dimensionality reduction and dimensionality increase matrices to obtain an optimized large language model.

[0140] The tuned large language model is adjusted through the autoregressive algorithm to obtain the target large language model.

[0141] In one embodiment, the step of pre-training the large language model based on the first training data set to obtain the pre-trained large language model includes:

[0142] Performing ViT feature encoding on the visual game image in the first training data set to obtain image encoding features;

[0143] and performing scene description coding on the quantifiable game features in the first training data set to obtain scene description coding features;

[0144] and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features;

[0145] A large language model is pre-trained based on the image coding features, the scene description coding features, and the task description coding features to obtain a pre-trained large language model.

[0146] In one embodiment, the anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, including:

[0147] determining a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0148] The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

[0149] In one embodiment, the initial policy behavior model is trained based on the output results to obtain a target policy behavior model, including:

[0150] Determine the contrast loss value based on the behavior pair to be evaluated and the behavior preference result through the contrast loss function;

[0151] Adjust the network parameters of the initial strategy behavior model based on the contrast loss value;

[0152] The proximal policy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0153] In one embodiment, the anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, and further includes:

[0154] determining a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0155] Through the target large language model, the behavior suggestion results matching the Prompt paradigm are output based on the behavior to be evaluated;

[0156] In one embodiment, the initial policy behavior model is trained based on the output results to obtain a target policy behavior model, further comprising:

[0157] Using the imitation learning algorithm, the initial strategy behavior model is trained based on the behavior recommendation results to obtain the target strategy behavior model.

[0158] The electronic device provided by the embodiment of the present disclosure uses a large language model to train a strategy behavior model, which can effectively improve the problems existing in the training process, such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm. The trained target strategy behavior model is applied to the anthropomorphism process, so as to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0159] Figure 6 is a structural diagram of an electronic device provided in an embodiment of the present disclosure. The electronic device 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63. The processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0160] The memory 61 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 63 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0161] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG6 shows only one bidirectional arrow, but this does not mean that there is only one bus or one type of bus.

[0162] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the aforementioned embodiment of the present disclosure can be applied to the processor 60 or implemented by the processor 60.

[0163] The processor 60 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 60 or software instructions. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 61, and processor 60 reads the information in memory 61 and, in conjunction with its hardware, completes the steps of the above method.

[0164] The computer program product of a readable storage medium provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, wherein the program code includes instructions that can be used to execute:

[0165] A behavior control method for a virtual character, comprising:

[0166] Obtaining a visual game image of the current game environment and quantifiable game features associated with a virtual character within the game environment;

[0167] Determining a target game behavior to be executed by the virtual character based on the visualized game image and the quantifiable game features using a pre-trained target strategy behavior model; wherein the target strategy behavior model is trained based on a large language model;

[0168] The virtual character is controlled to perform target game behaviors within the game environment so that the virtual character interacts with the game environment.

[0169] In one embodiment, the step of training the target policy behavior model includes:

[0170] Pre-training and adjusting the large language model based on a first training dataset to obtain a target large language model; wherein the first training dataset includes visual game images and quantifiable game features generated by players during game play;

[0171] Determining, using the initial strategic behavior model, a plurality of behavior segments to be evaluated based on a second training dataset, wherein the second training dataset includes a visual game image of an offline game environment and quantifiable game features associated with a virtual character within the game environment;

[0172] The target language model is used to determine the output based on a pre-built anthropomorphic prompt pattern and multiple behavior segments to be evaluated.

[0173] The initial strategy behavior model is trained based on the output results to obtain the target strategy behavior model.

[0174] In one embodiment, the step of pre-training and adjusting the large language model based on the first training dataset to obtain a target large language model includes:

[0175] Pre-training the large language model based on the first training data set to obtain a pre-trained large language model;

[0176] Fix the weight of the pre-trained large language model and optimize it through dimensionality reduction and dimensionality increase matrices to obtain an optimized large language model.

[0177] The tuned large language model is adjusted through the autoregressive algorithm to obtain the target large language model.

[0178] In one embodiment, the step of pre-training the large language model based on the first training data set to obtain the pre-trained large language model includes:

[0179] Performing ViT feature encoding on the visual game image in the first training data set to obtain image encoding features;

[0180] and performing scene description coding on the quantifiable game features in the first training data set to obtain scene description coding features;

[0181] and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features;

[0182] A large language model is pre-trained based on the image coding features, the scene description coding features, and the task description coding features to obtain a pre-trained large language model.

[0183] In one embodiment, the anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, including:

[0184] determining a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0185] The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

[0186] In one embodiment, the initial policy behavior model is trained based on the output results to obtain a target policy behavior model, including:

[0187] Determine the contrast loss value based on the behavior pair to be evaluated and the behavior preference result through the contrast loss function;

[0188] Adjust the network parameters of the initial strategy behavior model based on the contrast loss value;

[0189] The proximal policy optimization algorithm is used to perform anthropomorphic strategy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0190] In one embodiment, the anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the output result is determined by the target large language model based on the pre-built anthropomorphic prompt paradigm and multiple behavior segments to be evaluated, and further includes:

[0191] determining a behavior pair to be evaluated from a plurality of behavior segments to be evaluated;

[0192] Through the target large language model, the behavior suggestion results matching the Prompt paradigm are output based on the behavior to be evaluated;

[0193] In one embodiment, the initial policy behavior model is trained based on the output results to obtain a target policy behavior model, further comprising:

[0194] Using the imitation learning algorithm, the initial strategy behavior model is trained based on the behavior recommendation results to obtain the target strategy behavior model.

[0195] The readable storage medium provided by the embodiment of the present disclosure utilizes a large language model to train a strategy behavior model, which can effectively improve problems such as unstable data quality, complicated overall optimization process, low knowledge density of injected data, and low efficiency of training algorithm in the training process. The trained target strategy behavior model is applied to the anthropomorphism process, so as to utilize the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of the anthropomorphism.

[0196] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0197] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A method for controlling the behavior of a virtual character, comprising: Obtaining a visualized game image of a current game environment and quantifiable game features associated with a virtual character in the game environment; Determining the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features through the target strategy behavior model obtained through pre-training; wherein the target strategy behavior model is obtained through training based on the large language model; The virtual character is controlled to execute the target game behavior in the game environment so that the virtual character interacts with the game environment.

2. The method for controlling the behavior of a virtual character according to claim 1, wherein: The training steps of the target strategy behavior model include: Pre-training and adjusting the large language model based on a first training data set to obtain a target large language model; wherein the first training data set includes visualized game images and quantifiable game features generated during a player's game process; Determine, through the initial strategy behavior model, a plurality of behavior segments to be evaluated based on a second training data set; wherein the second training data set includes a visualized game image of an offline game environment and quantifiable game features associated with the virtual character located in the game environment; Determine the output result through the target large language model based on the pre-constructed anthropomorphic prompt paradigm and the multiple behavior segments to be evaluated; The initial strategy behavior model is trained based on the output result to obtain a target strategy behavior model.

3. The method for controlling the behavior of a virtual character according to claim 2, wherein: The step of pre-training and adjusting the large language model based on the first training data set to obtain a target large language model includes: Pre-training the large language model based on the first training data set to obtain a pre-trained large language model; The weight of the pre-trained large language model is fixed, and the pre-trained large language model is optimized by using a dimensionality reduction matrix and a dimensionality increase matrix to obtain an optimized large language model; The tuned large language model is adjusted by an autoregressive algorithm to obtain a target large language model.

4. The method for controlling the behavior of a virtual character according to claim 2, wherein: The step of pre-training the large language model based on the first training data set includes: Performing ViT feature encoding on the visualized game image in the first training data set to obtain image encoding features; and performing scene description encoding on the quantifiable game features in the first training data set to obtain scene description encoding features; and performing task description encoding on the quantifiable game features in the first training data set to obtain task description encoding features; The large language model is pre-trained based on the image coding features, the scene description coding features and the task description coding features to obtain a pre-trained large language model.

5. The method for controlling the behavior of a virtual character according to claim 2, wherein: The anthropomorphic prompt paradigm includes a preference prompt paradigm, and the output result includes a behavior preference result; the output result is determined by the target large language model based on the pre-constructed anthropomorphic prompt paradigm and the multiple behavior segments to be evaluated, including: Determining a pair of behaviors to be evaluated from a plurality of the behaviors to be evaluated; The preference prompt paradigm and the behavior pair to be evaluated are input into the target large language model, so that the target large language model outputs a behavior preference result.

6. The method for controlling the behavior of a virtual character according to claim 5, wherein: The initial strategy behavior model is trained based on the output result to obtain a target strategy behavior model, including: Determining a contrast loss value based on the behavior pair to be evaluated and the behavior preference result by using a contrast loss function; Adjusting the network parameters of the initial strategy behavior model based on the contrast loss value; Using the proximal strategy optimization algorithm, the adjusted initial strategy behavior model is optimized by anthropomorphic strategy, and the result is To the target strategy behavior model.

7. The method for controlling the behavior of a virtual character according to claim 2, wherein: The anthropomorphic prompt paradigm includes a suggestion prompt paradigm, and the output result includes a behavior suggestion result; the output result is determined by the target large language model based on the pre-constructed anthropomorphic prompt paradigm and the multiple behavior segments to be evaluated, and further includes: Determining a pair of behaviors to be evaluated from a plurality of the behaviors to be evaluated; Outputting, through the target large language model, a behavior suggestion result matching the suggestion-style Prompt paradigm based on the behavior to be evaluated; The initial strategy behavior model is trained based on the output result to obtain a target strategy behavior model, further comprising: The initial strategy behavior model is trained based on the behavior suggestion results using an imitation learning algorithm to obtain a target strategy behavior model.

8. A behavior control device for a virtual character, comprising: A data acquisition module is configured to acquire a visual game image of a current game environment and quantifiable game features associated with a virtual character in the game environment; A behavior determination module is configured to execute a target strategy behavior model obtained through pre-training, and determine a target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features; wherein the target strategy behavior model is obtained through training based on a large language model; The interaction module is configured to control the virtual character to perform the target game behavior in the game environment so that the virtual character interacts with the game environment.

9. An electronic device, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images