Virtual role behavior control method and device, electronic equipment and storage medium

Abstract

The application provides a virtual role behavior control method and device, electronic equipment and storage medium, relates to the technical field of games, and comprises the following steps: acquiring a visual game image of a game environment at a current moment and a quantifiable game feature associated with a virtual role in the game environment; determining a target game behavior to be executed by the virtual role based on the visual game image and the quantifiable game feature through a target strategy behavior model obtained by pre-training; wherein the target strategy behavior model is obtained based on a large language model; and controlling the virtual role to execute the target game behavior in the game environment, so that the virtual role interacts with the game environment. The application can significantly improve the stability of personification effect and the overall efficiency of personification.

Description

Technical Field

[0001] This invention relates to the field of game technology, and in particular to a method, device, electronic device, and storage medium for controlling the behavior of virtual characters. Background Technology

[0002] To achieve human-like policy generation, the reward function of reinforcement learning algorithms typically requires careful design. While meticulously adjusting the reward function can address the issue of some agents not closely resembling human behavior, the inherent subjectivity of anthropomorphism makes designing specific reward functions for agent behavior optimization difficult. Therefore, researchers often employ imitation learning methods, which are not based on reward functions, to generate human-like agent behavior. As an offline data optimization scheme, imitation learning only requires the historical behavior sequences of experts to learn expert policies, thus achieving objective imitation of expert behavior.

[0003] While imitation learning objectively achieves the learning of game behavior, the complexity and variability of human data make it difficult to effectively mimic detailed game behaviors. Therefore, most existing technologies perform secondary optimization of the strategies learned through imitation learning from a subjective perspective. Human feedback-based reinforcement learning is a reinforcement learning algorithm that optimizes the model's strategy by leveraging human subjective preferences. Specifically, it first interacts the strategy learned through imitation learning with the simulation environment, generating several sequence samples to be evaluated. Then, human experts score and evaluate the sequence samples. Based on the expert scores, the algorithm trains a corresponding reward function model and performs final, anthropomorphic policy optimization based on this reward model.

[0004] However, while this method based on active scoring by human experts can effectively improve the anthropomorphic performance of the model learning model, it still faces many challenges from a practical application perspective: (1) unstable data quality, heavily influenced by annotation experts; (2) a complex overall optimization process; and (3) low knowledge density in the labeled data, resulting in low efficiency of the training algorithm. These challenges lead to poor stability in the anthropomorphic effect and low overall efficiency in the anthropomorphic process. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, device, electronic device and storage medium for controlling the behavior of virtual characters, which can significantly improve the stability of anthropomorphism and significantly improve the overall efficiency of anthropomorphism.

[0006] In a first aspect, embodiments of the present invention provide a method for controlling the behavior of a virtual character, comprising:

[0007] Obtain a visualized game image of the game environment at the current moment and quantifiable game features associated with virtual characters located within the game environment;

[0008] Based on the visualized game image and the quantifiable game features, the target game behavior to be performed by the virtual character is determined using a pre-trained target strategy behavior model; wherein, the target strategy behavior model is trained based on a large language model.

[0009] The virtual character is controlled to perform the target game behavior within the game environment, so that the virtual character interacts with the game environment.

[0010] Secondly, embodiments of the present invention also provide a behavior control device for a virtual character, comprising:

[0011] The data acquisition module is used to acquire a visualized game image of the game environment at the current moment and quantifiable game features associated with virtual characters located in the game environment;

[0012] The behavior determination module is used to determine the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features, using a pre-trained target strategy behavior model; wherein, the target strategy behavior model is trained based on a large language model;

[0013] An interaction module is used to control the virtual character to perform the target game behavior within the game environment, so that the virtual character can interact with the game environment.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method described in any of the first aspects.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method described in any of the first aspects.

[0016] This invention provides a method, device, electronic device, and storage medium for controlling the behavior of a virtual character. First, it acquires a visualized game image of the game environment at the current moment and quantifiable game features associated with the virtual character within that environment. Then, using a target strategy behavior model trained based on a large language model, it determines the target game behavior to be performed by the virtual character based on the visualized game image and quantifiable game features. This allows the virtual character to be controlled to perform the target game behavior within the game environment, enabling interaction between the virtual character and the game environment. This method utilizes a large language model to train the strategy behavior model, effectively improving problems such as unstable data quality, complex overall optimization processes, low knowledge density of the injected data, and low training algorithm efficiency during training. Applying the trained target strategy behavior model to the anthropomorphism process allows the model to determine the required target game behavior based on the visualized game image and quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and the overall efficiency of anthropomorphism.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a method for controlling the behavior of a virtual character, provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram illustrating a large language model pre-training process provided in an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structure of a LoRA model provided in an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a preference-based prompt paradigm and a suggestion-based prompt paradigm provided in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the structure of a behavior control device for a virtual character provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Currently, while methods based on active scoring by human experts can effectively improve the human-like performance of learning models, there are still many challenges from a practical application perspective:

[0028] (1) Unstable data quality, heavily influenced by annotation experts: Unlike general image annotation, anthropomorphic style annotation requires experts to have a deep understanding of the anthropomorphic behavior of game NPCs, what human players behave like, and what kind of NPCs behave like humans. This places high constraints on the abilities of annotation experts. Furthermore, anthropomorphism itself is a subjective evaluation, and different experts are prone to bias in their anthropomorphic evaluations of different pedestrians. How to unify the anthropomorphic preferences of experts remains a difficult problem to solve.

[0029] (2) Complex overall optimization process: Since reinforcement learning based on human feedback is a human-in-the-loop optimization method, the entire method is heavily dependent on the annotation efficiency of experts. In order to ensure high-quality anthropomorphic optimization results, a large amount of data usually needs to be annotated, making the overall annotation and quality inspection process too complicated and inefficient.

[0030] (3) Low knowledge density of labeled data leads to low training algorithm efficiency: Existing methods only optimize game behavior based on preference-based labeling results, i.e., experts select the more human-like behavior segment by comparing two NPC behavior segments. Compared to selecting the more human-like segment, providing suggestions on how to operate in a more human-like manner is more dense, but this method places too high demands on the labeling experts and is often difficult to implement. Game behavior optimization based on preference-based labeling results is difficult to achieve high training efficiency due to the low knowledge density of the training data.

[0031] Based on this, the present invention provides a method, device, electronic device and storage medium for controlling the behavior of virtual characters, which can significantly improve the stability of anthropomorphism and significantly improve the overall efficiency of anthropomorphism.

[0032] To facilitate understanding of this embodiment, a method for controlling the behavior of a virtual character disclosed in this invention will first be described in detail. (See also...) Figure 1 The diagram shows a flowchart of a method for controlling the behavior of a virtual character. The method mainly includes the following steps S102 to S106:

[0033] Step S102: Obtain the visualized game image of the game environment at the current moment and the quantifiable game features associated with the virtual characters located in the game environment.

[0034] Among them, the visual game image is the game screen provided to the user through the graphical user interface of the terminal, which may include the game scene and minimap that are visible to the player; the virtual character can be an NPC (non-player character); the quantifiable game features are game data with specific values, including the attributes and characteristics of NPCs and related environmental information. The attribute information includes the NPC's health, position, speed, attack power, buffs, etc., and the environmental information includes time, map information, the position and speed information of other NPCs, etc.

[0035] Step S104: Using the pre-trained target strategy behavior model, based on visualized game images and quantifiable game features, determine the target game behavior to be performed by the virtual character. The target strategy behavior model is trained based on a large language model.

[0036] In one implementation, the aforementioned visualized game image and quantifiable game features are input into the target strategy behavior model to obtain the target game behavior output by the target strategy behavior model.

[0037] In practical applications, the initial strategy behavior model can be pre-trained to obtain the desired target strategy behavior model. During the training process, the pre-trained target large language model and the pre-built anthropomorphic Prompt paradigm can be used to guide the output of the initial strategy behavior model, thereby training the initial strategy behavior model and obtaining the target strategy behavior model.

[0038] Step S106: Control the virtual character to perform the target game behavior within the game environment, so that the virtual character can interact with the game environment.

[0039] The target game behavior, which simulates the game behavior of a real player, can also be called anthropomorphic behavior, such as moving forward, retreating, attacking, defending, and interacting with other virtual objects. For example, assuming the target game behavior is moving forward, the virtual character will be controlled to move forward within the game environment.

[0040] The virtual character behavior control method provided in this invention utilizes a large language model to train a strategy behavior model, which can effectively improve problems such as unstable data quality, complex overall optimization process, low knowledge density of data, and low efficiency of training algorithms during the training process. The trained target strategy behavior model is applied to the anthropomorphism process, and the target game behavior is determined based on the visualized game image and the quantifiable game features using the target strategy behavior model. This can significantly improve the stability of the anthropomorphism effect and significantly improve the overall efficiency of anthropomorphism.

[0041] The main problem addressed by this invention is how to efficiently perform subjective annotation of NPC game behaviors and thereby optimize the training efficiency of the algorithm. Specifically, this invention addresses the following challenges: (1) Subjective anthropomorphic annotation not based on human-in-the-loop: Human-in-the-loop anthropomorphic data annotation suffers from unstable data quality and complex data acquisition and quality inspection processes. Anthropomorphic data annotation is a subjective style annotation, and traditional scripting schemes struggle to achieve subjective recognition of anthropomorphic styles. Therefore, a subjective anthropomorphic annotation not based on human-in-the-loop is needed to significantly improve the efficiency and effectiveness of subjective anthropomorphic style annotation. (2) Anthropomorphic optimization algorithm based on non-expert annotation: Traditional preference-based anthropomorphic style optimization schemes only optimize game behaviors using preference data from the behavior trajectories of two NPCs, resulting in low overall training efficiency. Therefore, exploring a more efficient anthropomorphic training scheme is essential.

[0042] To address the above problems, this invention proposes a method for training a target strategy behavior model, which mainly consists of: (1) a language model Finetune that combines video segment encoding based on ViT Embedding with semantic information of the game environment; (2) NPC anthropomorphic prompt construction; and (3) NPC game behavior optimization based on the response of a large language model.

[0043] For ease of understanding, this embodiment of the invention provides an implementation method for training a target policy behavior model, as detailed in steps 1 to 4 below:

[0044] Step 1: Pre-train and adjust the large language model based on the first training dataset to obtain the target large language model. The first training dataset includes visual game images and quantifiable game features generated during gameplay, i.e., real human player data.

[0045] This invention proposes a language model called Finetune, which combines video segment encoding based on ViT Embedding with semantic information from the game environment. Its purpose is to optimize the understanding of games by a large language model, enabling it to more accurately comprehend the game behavior of NPCs. Unlike traditional methods that focus solely on fine-tuning the large language model itself, this invention draws inspiration from PaLM-E in robot control, introducing Vision-Transformer (ViT) to extract game features from actual game footage, thereby enhancing the large language model's ability to understand games. Specifically, model pre-training includes constructing a pre-training dataset, optimizing an LLM model combining ViT and Low-Rank algorithms, and training the Finetune model.

[0046] For ease of understanding, this embodiment of the invention provides an implementation method for pre-training and adjusting a large language model based on a first training dataset to obtain a target large language model, as shown in steps 1.1 to 1.3 below:

[0047] Step 1.1: Pre-train the large language model based on the first training dataset to obtain the pre-trained large language model. In specific implementation, to improve the model's understanding of human behavior in games, this embodiment uses real human player data to construct the pre-training dataset. This embodiment records the game screen of real game players, game state features that can be vectorized, text features processed by scene description units and task description units, and the actual actions performed by players. When constructing the output of the large language model, compared to directly using human actions, this embodiment maps player actions to text commands. For example, if the player's action is to attack the enemy to the right front, this embodiment maps this action to the text feature "Facing the current state, select the right front as the execution target, and selecting work as the execution action is the most appropriate".

[0048] Typical large language models can only process textual features. Therefore, this invention introduces the Vision-Transformer (ViT) network to process game image features. For details, see... Figure 2 The diagram shown illustrates a pre-training process for a large language model, including the following (1) to (2):

[0049] (1) ViT feature encoding is performed on the visualized game images in the first training dataset to obtain image encoding features; scene description encoding is performed on the quantifiable game features in the first training dataset to obtain scene description encoding features; and task description encoding is performed on the quantifiable game features in the first training dataset to obtain task description encoding features.

[0050] In practical applications, the process begins by acquiring the current visual game image (i.e., image features) and related quantifiable game features from the game environment, such as environment state and NPC state. Then, vector encoding is performed on both the visual game image and the quantifiable game features. Specifically, the visual game image undergoes feature encoding based on a ViT network to divide the complete game image into several game sub-images (i.e., image-encoded features); the quantifiable game features are then encoded using scene description and task description, resulting in scene description encoded features and task description encoded features.

[0051] For example, please continue to see Figure 2 , Figure 2 Examples illustrating scene description coding features include "You see 2 opponents", "Opponent 1 is 2 meters to your right front", "Opponent 2 is 3 meters to your left front", "The opponent is holding a halberd", etc. Figure 2 It also illustrates examples of task description coding features, including "Scenario 1: You previously chose to attack the person to your left front. If you were a human player, what would you consider doing?" and "Scenario 2: Faced with this situation, you did not take any task actions before. What would you do now?"

[0052] (2) The large language model is pre-trained based on image coding features, scene description coding features and task description coding features to obtain the pre-trained large language model.

[0053] In practical applications, the game subgraph is encoded using neural network vectors and input into a large language model to aid in understanding the game state. Simultaneously, scene description encoding features and task description encoding features are input into the large language model. The large language model responds by outputting behavioral description text, which is then transformed into executable NPC actions, thereby enabling NPC environmental interaction.

[0054] For example, please continue to see Figure 2 , Figure 2 The behavioral description text output by the large language model may be "In case 1, I will switch to attacking the opponent who is relatively close to the right front" or "In case 2, I will directly attack the enemy to the right front because he is closer".

[0055] Furthermore, the large language model is pre-trained based on its output and the actual behavior of real human players.

[0056] Step 1.2: Fix the weights of the pre-trained large language model, and fine-tune the pre-trained large language model using dimensionality reduction and dimensionality increase matrices to obtain the fine-tuned large language model.

[0057] Optionally, Low-Rank (LoRA) is a classic pre-trained model optimization scheme; see [link to relevant documentation]. Figure 3 The diagram illustrates the structure of a LoRA model. Compared to the traditional Finetune approach, which optimizes the parameters of the pre-trained model itself, LoRA fixes the weights of the pre-trained model and fine-tunes the model by optimizing the dimensionality reduction matrix A and the dimensionality increase matrix B. This optimization scheme can effectively reduce the parameters of the pre-trained model while ensuring training effectiveness.

[0058] Step 1.3: Adjust the optimized large language model using an autoregressive algorithm to obtain the target large language model.

[0059] Optionally, during the model fineetune training phase, this embodiment of the invention employs a classic autoregressive approach to optimize the model fineetune.

[0060] Step 2: Using the initial strategy behavior model, determine multiple behavior segments to be evaluated based on the second training dataset. The second training dataset includes visualized game images of the game environment in offline mode and quantifiable game features associated with virtual characters within the game environment.

[0061] In one implementation, the visualized game image of the game environment in the offline state and the quantifiable game features associated with NPCs are input into the initial strategy behavior model. The initial strategy behavior model will output multiple game behaviors, that is, multiple segments of behavior to be evaluated.

[0062] Step 3: Using the target large language model, determine the output based on the pre-built anthropomorphic Prompt paradigm and multiple segments of behavior to be evaluated. The anthropomorphic Prompt paradigm includes a preference-based prompt paradigm and a suggestion-based prompt paradigm, and the output includes behavioral preference results and behavioral suggestion results.

[0063] After completing the rendering of the large language model, in order to enable the model to output more appropriate game behavior suggestions, the solution then proceeds to construct an NPC-style anthropomorphic prompt. To fully utilize the understanding capabilities of the large language model and broaden the knowledge density of the labeled data, the NPC-style anthropomorphic prompt construction phase will construct preference-based prompt paradigms and suggestion-based prompt paradigms based on the characteristics of the questions generated by anthropomorphic decision-making behavior.

[0064] See Figure 4 The diagram illustrates a preference-based prompt paradigm and a suggestion-based prompt paradigm. Figure 4 Examples of a preference-based prompt paradigm and a suggestion-based prompt paradigm are provided respectively.

[0065] In one example, the preference prompt paradigm can be described as follows:

[0066] "You are now an expert in evaluating the anthropomorphism of game NPCs. Please evaluate the following two scenarios and select the decision-making behavior that is most human-like."

[0067] Scene A: 1. You see two opponents; 2. Opponent 1 is 2 meters to your right front; 3. Opponent 2 is 3 meters to your left front; 4. The opponents are wielding halberds.

[0068] The NPC's decision-making target is Opponent 1, and its decision-making behavior is to attack;

[0069] Scene A: 1. You see two opponents; 2. Opponent 1 is 2 meters to your right front; 3. Opponent 2 is 3 meters to your left front; 4. The opponents are wielding halberds.

[0070] The NPC's decision-making target is opponent 2, and its decision-making behavior is to attack;

[0071] As a personified evaluation expert, what do you think is the most appropriate segment?

[0072] In one example, the suggestion paradigm is described as follows:

[0073] "You are now an expert in evaluating the anthropomorphism of game NPCs. Please provide suggestions on the following NPC decision-making process."

[0074] 1. You see two opponents; 2. Opponent 1 is 2 meters to your right front; 3. Opponent 2 is 3 meters to your left front; 4. The opponent is wielding a halberd.

[0075] As a personified evaluation expert, what do you think are the most appropriate decision-making targets and decision-making behaviors for the NPC at this moment?

[0076] exist Figure 4 Based on this, embodiments of the present invention provide implementation methods for determining the output results of the target large language model based on the preference-based prompt paradigm and the suggestion-based prompt paradigm, respectively, as shown in Method 1 to Method 2 below:

[0077] Method 1, based on the preference-based prompt paradigm: The preference-based prompt paradigm is a preference judgment scheme. Its input is two behavioral segments to be judged, and the output is the judgment result of human experts or models. Specifically, the behavioral pairs to be evaluated can be determined from multiple behavioral segments to be evaluated. Then, the preference-based prompt paradigm and the behavioral pairs to be evaluated are input into the target large language model so that the target large language model outputs behavioral preference results.

[0078] In this embodiment of the invention, the initial policy behavior model, pre-trained through reinforcement learning or imitation learning, interacts with the game environment to generate multiple behavior segments to be evaluated. Subsequently, similar behavior segments are selected using manual rules to construct behavior pairs to be evaluated. Finally, the behavior pairs to be evaluated and the preference prompt paradigm are input into a large language model to obtain the behavior preference results.

[0079] Method 2, based on the suggestion-based prompt paradigm: The suggestion-based prompt paradigm aims to enable the large language model to directly propose reasonable behavioral suggestions based on the current behavior of the NPC. Specifically, it can identify pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated; the target large language model then outputs behavioral suggestions matching the suggestion-based prompt paradigm based on the pairs of behaviors to be evaluated. For example, the pairs of behaviors to be evaluated can be input into the target large language model so that the target large oracle model outputs behavioral suggestions matching the suggestion-based prompt paradigm.

[0080] Step 4: Train the initial strategy behavior model based on the output results to obtain the target strategy behavior model. In one implementation, after obtaining the behavior preference results and behavior suggestion results of the language model, the NPC game behavior optimization based on the large language model response will apply these results to the game behavior optimization of the NPC strategy behavior model.

[0081] In one example, based on the behavioral preference results output by the large language model, the initial policy behavior model can be trained according to the following steps a to b:

[0082] Step a involves determining the contrastive loss value based on the behavior pair to be evaluated and the behavior preference results using a contrastive loss function, and then adjusting the network parameters of the initial policy behavior model based on the contrastive loss value.

[0083] In practical implementation, a reinforcement learning method based on human feedback can be used to optimize game behavior. This method includes two stages: reward model training and game behavior optimization based on the reward function. During reward model training, the scheme constructs the model by building a value function branch on a large language model completed with Finetune, and optimizes the reward model by comparing the loss function, as shown below:

[0084]

[0085] Where x and y represent NPC behavior fragments and behavior preference results, respectively, and r θ σ represents the value of the reward model with parameter θ, and σ represents the sigmoid activation function.

[0086] Step b: Using the proximal policy optimization algorithm, perform anthropomorphic policy optimization on the adjusted initial policy behavior model to obtain the target policy behavior model.

[0087] In its specific implementation, after completing the training of the reward function, this embodiment of the invention employs a reinforcement learning algorithm based on proximal policy optimization to perform anthropomorphic policy optimization, thereby improving game behavior.

[0088] In one example, based on the behavior suggestion results output by the large language model, the initial strategy behavior model can be trained according to the following steps: using an imitation learning algorithm, the initial strategy behavior model is trained based on the behavior suggestion results to obtain the target strategy behavior model. In specific implementation, this embodiment of the invention uses imitation learning to improve the model's game behavior.

[0089] In summary, the virtual character behavior control method provided in this embodiment of the invention has at least the following characteristics:

[0090] (1) Compared with human crowdsourced annotation, Finetune, a language model that combines video segment encoding based on ViT Embedding with semantic information of the game environment, can achieve a comprehensive understanding of the human-like task objectives based on the understanding ability of a large language model and human preferences, and fully grasp the behavioral tendencies of human players. Annotating and correcting NPC behavior through language models can effectively avoid problems such as unstable data quality and complicated data acquisition and quality inspection processes.

[0091] (2) To fully utilize the understanding capabilities of large language models and broaden the knowledge density of labeled data, the NPC anthropomorphic prompt construction stage will construct preference-based prompts and suggestion-based prompts based on the characteristics of the questions generated by anthropomorphic decision-making behavior. The construction of multiple anthropomorphic prompts can provide a data foundation for improving the efficiency of anthropomorphic training in the future.

[0092] (3) Based on preference-based prompts and suggestion-based prompts, the NPC game behavior optimization method based on large language model response can significantly improve the optimization efficiency of game behavior and the final anthropomorphic effect through a joint optimization method combining reinforcement learning and imitation learning.

[0093] Based on the foregoing embodiments, this invention provides a behavior control device for virtual characters, see [link to previous embodiment]. Figure 5 The diagram shows a structural schematic of a behavior control device for a virtual character. The device mainly includes the following parts:

[0094] The data acquisition module 502 is used to acquire the visualized game image of the game environment at the current moment and the quantifiable game features associated with the virtual characters located in the game environment;

[0095] The behavior determination module 504 is used to determine the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features, using a pre-trained target strategy behavior model; wherein, the target strategy behavior model is trained based on a large language model;

[0096] The interaction module 506 is used to control the virtual character to perform target game behaviors within the game environment, so that the virtual character can interact with the game environment.

[0097] The virtual character behavior control device provided in this embodiment of the invention utilizes a large language model to train a strategy behavior model, which can effectively improve problems such as unstable data quality, complex overall optimization process, low knowledge density of data, and low efficiency of training algorithms during the training process. The trained target strategy behavior model is applied to the anthropomorphism process, so as to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of anthropomorphism.

[0098] In one implementation, a model training module is also included, for:

[0099] The target large language model is obtained by pre-training and adjusting the large language model based on the first training dataset; the first training dataset includes visual game images and quantifiable game features generated by players during gameplay (i.e., while the game is online);

[0100] Using the initial strategy behavior model, multiple behavior segments to be evaluated are determined based on the second training dataset; the second training dataset includes visualized game images of the game environment in offline state and quantifiable game features associated with virtual characters located in the game environment;

[0101] The output is determined by using a target large language model based on a pre-built anthropomorphic Prompt paradigm and multiple segments of behavior to be evaluated.

[0102] The initial policy behavior model is trained based on the output results to obtain the target policy behavior model.

[0103] In one implementation, the model training module is further used for:

[0104] The large language model is pre-trained based on the first training dataset to obtain the pre-trained large language model.

[0105] The weights of the pre-trained large language model are fixed, and the pre-trained large language model is tuned by dimensionality reduction matrix and dimensionality increase matrix to obtain the tuned large language model.

[0106] The target large language model is obtained by adjusting the optimized large language model using an autoregressive algorithm.

[0107] In one implementation, the model training module is further used for:

[0108] ViT feature encoding is performed on the visualized game images in the first training dataset to obtain image encoding features;

[0109] And scene description encoding features are obtained by performing scene description encoding on the quantifiable game features in the first training dataset;

[0110] And the quantifiable game features in the first training dataset are subjected to task description encoding to obtain task description encoded features;

[0111] The large language model is pre-trained based on the image encoding features, the scene description encoding features, and the task description encoding features to obtain the pre-trained large language model.

[0112] In one implementation, the anthropomorphic Prompt paradigm includes a preference-based Prompt paradigm, and the output includes behavioral preference results; the model training module is also used for:

[0113] Identify the pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated;

[0114] The preference Prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference results.

[0115] In one implementation, the model training module is further used for:

[0116] The comparative loss value is determined by comparing the loss function based on the behavior pair to be evaluated and the behavior preference results.

[0117] The network parameters of the initial policy behavior model are adjusted based on the contrastive loss value;

[0118] By using the proximal policy optimization algorithm, the adjusted initial policy behavior model is optimized using anthropomorphic policy to obtain the target policy behavior model.

[0119] In one implementation, the anthropomorphic Prompt paradigm includes a suggestion-based Prompt paradigm, and the output includes behavioral suggestion results; the model training module is also used for:

[0120] Identify the pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated;

[0121] Using a target large language model, behavioral suggestion results are obtained by matching the output suggestion Prompt paradigm based on the behavior to be evaluated.

[0122] In one implementation, the model training module is further used for:

[0123] By using an imitation learning algorithm, the initial policy behavior model is trained based on the behavior suggestion results to obtain the target policy behavior model.

[0124] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0125] This invention provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, which is executed by the processor when it runs:

[0126] A method for controlling the behavior of a virtual character, comprising:

[0127] Obtain a visual game image of the game environment at the current moment and quantifiable game features associated with virtual characters located within the game environment;

[0128] Based on the visualized game images and quantifiable game features, the target game behavior to be performed by the virtual character is determined using a pre-trained target strategy behavior model; wherein, the target strategy behavior model is trained based on a large language model.

[0129] Control the virtual character to perform target game behaviors within the game environment, so that the virtual character can interact with the game environment.

[0130] In one implementation, the training steps of the target policy behavior model include:

[0131] The target large language model is obtained by pre-training and adjusting the large language model based on the first training dataset; the first training dataset includes visual game images and quantifiable game features generated by players during gameplay.

[0132] Using the initial strategy behavior model, multiple behavior segments to be evaluated are determined based on the second training dataset; the second training dataset includes visualized game images of the game environment in offline state and quantifiable game features associated with virtual characters located in the game environment;

[0133] The output is determined by using a target large language model based on a pre-built anthropomorphic Prompt paradigm and multiple segments of behavior to be evaluated.

[0134] The initial policy behavior model is trained based on the output results to obtain the target policy behavior model.

[0135] In one implementation, the step of pre-training and adjusting a large language model based on a first training dataset to obtain a target large language model includes:

[0136] The large language model is pre-trained based on the first training dataset to obtain the pre-trained large language model.

[0137] The weights of the pre-trained large language model are fixed, and the pre-trained large language model is tuned by dimensionality reduction matrix and dimensionality increase matrix to obtain the tuned large language model.

[0138] The target large language model is obtained by adjusting the optimized large language model using an autoregressive algorithm.

[0139] In one implementation, the step of pre-training a large language model based on a first training dataset to obtain a pre-trained large language model includes:

[0140] ViT feature encoding is performed on the visualized game images in the first training dataset to obtain image encoding features;

[0141] And scene description encoding features are obtained by performing scene description encoding on the quantifiable game features in the first training dataset;

[0142] And the quantifiable game features in the first training dataset are subjected to task description encoding to obtain task description encoded features;

[0143] The large language model is pre-trained based on the image encoding features, the scene description encoding features, and the task description encoding features to obtain the pre-trained large language model.

[0144] In one implementation, the anthropomorphic Prompt paradigm includes a preference-based Prompt paradigm, and the output includes behavioral preference results. The output is determined using a target large language model based on a pre-built anthropomorphic Prompt paradigm and multiple behavioral segments to be evaluated, including:

[0145] Identify the pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated;

[0146] The preference Prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference results.

[0147] In one implementation, an initial policy behavior model is trained based on the output results to obtain a target policy behavior model, including:

[0148] The comparative loss value is determined by comparing the loss function based on the behavior pair to be evaluated and the behavior preference results.

[0149] The network parameters of the initial policy behavior model are adjusted based on the contrastive loss value;

[0150] By using the proximal policy optimization algorithm, the adjusted initial policy behavior model is optimized using anthropomorphic policy to obtain the target policy behavior model.

[0151] In one implementation, the anthropomorphic Prompt paradigm includes a suggestion-based Prompt paradigm, and the output includes behavioral suggestion results; the output results are determined based on a pre-built anthropomorphic Prompt paradigm and multiple behavioral segments to be evaluated using a target large language model, and further include:

[0152] Identify the pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated;

[0153] Using a target large language model, behavioral suggestion results are obtained by matching the output suggestion Prompt paradigm based on the behavior to be evaluated.

[0154] In one implementation, the initial policy behavior model is trained based on the output results to obtain the target policy behavior model, and the method further includes:

[0155] By using an imitation learning algorithm, the initial policy behavior model is trained based on the behavior suggestion results to obtain the target policy behavior model.

[0156] The electronic device provided in this invention utilizes a large language model to train a strategy behavior model, which can effectively improve problems such as unstable data quality, complex overall optimization process, low knowledge density of data, and low efficiency of training algorithms during the training process. The trained target strategy behavior model is applied to the anthropomorphism process to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of anthropomorphism.

[0157] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes: a processor 60, a memory 61, a bus 62, and a communication interface 63. The processor 60, the communication interface 63, and the memory 61 are connected through the bus 62. The processor 60 is used to execute executable modules, such as computer programs, stored in the memory 61.

[0158] The memory 61 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 63 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0159] Bus 62 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0160] The memory 61 is used to store programs. After receiving an execution instruction, the processor 60 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0161] Processor 60 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 60 or by instructions in software form. Processor 60 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 61. Processor 60 reads the information in memory 61 and, in conjunction with its hardware, completes the steps of the above method.

[0162] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code, wherein the program code includes instructions that can be executed:

[0163] A method for controlling the behavior of a virtual character, comprising:

[0164] Obtain a visual game image of the game environment at the current moment and quantifiable game features associated with virtual characters located within the game environment;

[0165] Based on the visualized game images and quantifiable game features, the target game behavior to be performed by the virtual character is determined using a pre-trained target strategy behavior model; wherein, the target strategy behavior model is trained based on a large language model.

[0166] Control the virtual character to perform target game behaviors within the game environment, so that the virtual character can interact with the game environment.

[0167] In one implementation, the training steps of the target policy behavior model include:

[0168] The target large language model is obtained by pre-training and adjusting the large language model based on the first training dataset; the first training dataset includes visual game images and quantifiable game features generated by players during gameplay.

[0169] Using the initial strategy behavior model, multiple behavior segments to be evaluated are determined based on the second training dataset; the second training dataset includes visualized game images of the game environment in offline state and quantifiable game features associated with virtual characters located in the game environment;

[0170] The output is determined by using a target large language model based on a pre-built anthropomorphic Prompt paradigm and multiple segments of behavior to be evaluated.

[0171] The initial policy behavior model is trained based on the output results to obtain the target policy behavior model.

[0172] In one implementation, the step of pre-training and adjusting a large language model based on a first training dataset to obtain a target large language model includes:

[0173] The large language model is pre-trained based on the first training dataset to obtain the pre-trained large language model.

[0174] The weights of the pre-trained large language model are fixed, and the pre-trained large language model is tuned by dimensionality reduction matrix and dimensionality increase matrix to obtain the tuned large language model.

[0175] The target large language model is obtained by adjusting the optimized large language model using an autoregressive algorithm.

[0176] In one implementation, the step of pre-training a large language model based on a first training dataset to obtain a pre-trained large language model includes:

[0177] ViT feature encoding is performed on the visualized game images in the first training dataset to obtain image encoding features;

[0178] And scene description encoding features are obtained by performing scene description encoding on the quantifiable game features in the first training dataset;

[0179] And the quantifiable game features in the first training dataset are subjected to task description encoding to obtain task description encoded features;

[0180] The large language model is pre-trained based on the image encoding features, the scene description encoding features, and the task description encoding features to obtain the pre-trained large language model.

[0181] In one implementation, the anthropomorphic Prompt paradigm includes a preference-based Prompt paradigm, and the output includes behavioral preference results. The output is determined using a target large language model based on a pre-built anthropomorphic Prompt paradigm and multiple behavioral segments to be evaluated, including:

[0182] Identify the pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated;

[0183] The preference Prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference results.

[0184] In one implementation, an initial policy behavior model is trained based on the output results to obtain a target policy behavior model, including:

[0185] The comparative loss value is determined by comparing the loss function based on the behavior pair to be evaluated and the behavior preference results.

[0186] The network parameters of the initial policy behavior model are adjusted based on the contrastive loss value;

[0187] By using the proximal policy optimization algorithm, the adjusted initial policy behavior model is optimized using anthropomorphic policy to obtain the target policy behavior model.

[0188] In one implementation, the anthropomorphic Prompt paradigm includes a suggestion-based Prompt paradigm, and the output includes behavioral suggestion results; the output results are determined based on a pre-built anthropomorphic Prompt paradigm and multiple behavioral segments to be evaluated using a target large language model, and further include:

[0189] Identify the pairs of behaviors to be evaluated from multiple segments of behaviors to be evaluated;

[0190] Using a target large language model, behavioral suggestion results are obtained by matching the output suggestion Prompt paradigm based on the behavior to be evaluated.

[0191] In one implementation, the initial policy behavior model is trained based on the output results to obtain the target policy behavior model, and the method further includes:

[0192] By using an imitation learning algorithm, the initial policy behavior model is trained based on the behavior suggestion results to obtain the target policy behavior model.

[0193] The readable storage medium provided in this embodiment of the invention utilizes a large language model to train a strategy behavior model, which can effectively improve problems such as unstable data quality, complex overall optimization process, low knowledge density of the data, and low efficiency of the training algorithm during the training process. The trained target strategy behavior model is applied to the anthropomorphism process, so as to use the target strategy behavior model to determine the required target game behavior based on the visualized game image and the quantifiable game features, thereby significantly improving the stability of the anthropomorphism effect and significantly improving the overall efficiency of anthropomorphism.

[0194] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling the behavior of a virtual character, characterized in that, include: Obtain a visualized game image of the game environment at the current moment and quantifiable game features associated with virtual characters located within the game environment; Based on the visualized game image and the quantifiable game features, the target game behavior to be performed by the virtual character is determined by the pre-trained target strategy behavior model. The virtual character is controlled to perform the target game behavior within the game environment, so that the virtual character interacts with the game environment; The training steps of the target policy behavior model include: The target large language model is obtained by pre-training and adjusting the large language model based on the first training dataset; wherein, the first training dataset includes visual game images and quantifiable game features generated by players during the game. Using an initial strategy behavior model, multiple behavior segments to be evaluated are determined based on a second training dataset; wherein, the second training dataset includes visualized game images of the game environment in an offline state and quantifiable game features associated with the virtual character located in the game environment; The output result is determined based on the target large language model, the pre-built anthropomorphic Prompt paradigm, and multiple segments of the behavior to be evaluated. The initial policy behavior model is trained based on the output results to obtain the target policy behavior model.

2. The virtual character behavior control method according to claim 1, characterized in that, The steps of pre-training and adjusting the large language model based on the first training dataset to obtain the target large language model include: The large language model is pre-trained based on the first training dataset to obtain the pre-trained large language model. The weights of the pre-trained large language model are fixed, and the pre-trained large language model is tuned using a dimensionality reduction matrix and an dimensionality increase matrix to obtain the tuned large language model. The optimized large language model is adjusted using an autoregressive algorithm to obtain the target large language model.

3. The method for controlling the behavior of virtual characters according to claim 1, characterized in that, The steps for pre-training the large language model based on the first training dataset include: ViT feature encoding is performed on the visualized game images in the first training dataset to obtain image encoding features; And scene description encoding features are obtained by performing scene description encoding on the quantifiable game features in the first training dataset; And the quantifiable game features in the first training dataset are subjected to task description encoding to obtain task description encoded features; The large language model is pre-trained based on the image encoding features, the scene description encoding features, and the task description encoding features to obtain the pre-trained large language model.

4. The virtual character behavior control method according to claim 1, characterized in that, The anthropomorphic Prompt paradigm includes a preference-based Prompt paradigm, and the output includes behavioral preference results. The output is determined using the target large language model based on the pre-constructed anthropomorphic Prompt paradigm and multiple segments of the behavior to be evaluated, including: Identify the pairs of behaviors to be evaluated from the multiple segments of behaviors to be evaluated; The preference Prompt paradigm and the behavior pair to be evaluated are input into the target large language model so that the target large language model outputs the behavior preference result.

5. The virtual character behavior control method according to claim 4, characterized in that, The initial policy behavior model is trained based on the output results to obtain the target policy behavior model, including: The comparative loss value is determined based on the pair of behaviors to be evaluated and the behavior preference results by comparing the loss function. The network parameters of the initial policy behavior model are adjusted based on the contrastive loss value. Using a proximal policy optimization algorithm, the adjusted initial policy behavior model is subjected to anthropomorphic policy optimization to obtain the target policy behavior model.

6. The method for controlling the behavior of virtual characters according to claim 1, characterized in that, The anthropomorphic Prompt paradigm includes a suggestion-based Prompt paradigm, and the output includes behavioral suggestion results; the output is determined based on the pre-built anthropomorphic Prompt paradigm and multiple segments of the behavior to be evaluated through the target large language model, and also includes: Identify the pairs of behaviors to be evaluated from the multiple segments of behaviors to be evaluated; Based on the target large language model, the behavioral suggestion result of the suggestion Prompt paradigm matching is output according to the behavior to be evaluated; The initial policy behavior model is trained based on the output results to obtain the target policy behavior model, which further includes: Using an imitation learning algorithm, the initial policy behavior model is trained based on the behavior suggestion results to obtain the target policy behavior model.

7. A behavior control device for a virtual character, characterized in that, include: The data acquisition module is used to acquire a visualized game image of the game environment at the current moment and quantifiable game features associated with virtual characters located in the game environment; The behavior determination module is used to determine the target game behavior to be performed by the virtual character based on the visualized game image and the quantifiable game features, using a pre-trained target strategy behavior model. An interaction module is used to control the virtual character to perform the target game behavior within the game environment, so that the virtual character can interact with the game environment; It also includes a model training module, used for: The target large language model is obtained by pre-training and adjusting the large language model based on the first training dataset; wherein, the first training dataset includes visual game images and quantifiable game features generated by players during the game. Using an initial strategy behavior model, multiple behavior segments to be evaluated are determined based on a second training dataset; wherein, the second training dataset includes visualized game images of the game environment in an offline state and quantifiable game features associated with the virtual character located in the game environment; The output result is determined based on the target large language model, the pre-built anthropomorphic Prompt paradigm, and multiple segments of the behavior to be evaluated. The initial policy behavior model is trained based on the output results to obtain the target policy behavior model.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images