Virtual character control method, device, electronic device, computer-readable storage medium and computer program product

Abstract

The present application provides a control method, device, electronic device, computer-readable storage medium and computer program product for a virtual character; the method comprises: obtaining a first state parameter of a first virtual character in a virtual scene, and a second state parameter of a second virtual character interacting with the first virtual character; determining a parameter difference between the first state parameter and the second state parameter; making an action decision based on the parameter difference to obtain a target action of the first virtual character; and controlling the first virtual character to perform the target action. Through the present application, the difficulty of making action decisions for virtual characters in a virtual scene can be reduced, thereby improving the efficiency of action decisions.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for controlling a virtual character. Background Art

[0002] In the related art, a behavior tree (BT) structure is usually used to implement the behavior logic decision of non-player characters (NPC), wherein the behavior tree is a tool for implementing complex behaviors of NPCs. When controlling NPCs to perform actions, the computer device traverses the behavior tree from the root node in the order of execution until the terminal state is reached. In the process of traversing the behavior tree, the computer device determines the next node to be executed based on the status information returned by different leaf nodes (such as success, failure, or running) based on the set rules, thereby implementing the behavior logic decision of the NPC.

[0003] However, the behavior tree structure is a rule-based algorithm. If the set rules remain unchanged, the NPC's performance in the game scene will be too simple. In order to make the NPC's performance in the game scene more diverse, it is usually necessary to formulate complex rules, which leads to greater production difficulty and time cost. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for controlling a virtual character, which can reduce the difficulty of making action decisions for a virtual character in a virtual scene, thereby improving the efficiency of action decisions.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for controlling a virtual character, including:

[0007] Acquire a first state parameter of a first virtual character in a virtual scene, and a second state parameter of a second virtual character interacting with the first virtual character;

[0008] determining a parameter difference between the first state parameter and the second state parameter;

[0009] Performing an action decision based on the parameter difference to obtain a target action of the first virtual character;

[0010] Control the first virtual character to perform the target action.

[0011] The present application provides a control device for a virtual character, including:

[0012] An acquisition module, used to acquire a first state parameter of a first virtual character in a virtual scene, and a second state parameter of a second virtual character interacting with the first virtual character;

[0013] a determination module, configured to determine a parameter difference between the first state parameter and the second state parameter;

[0014] an action decision module, configured to make an action decision based on the parameter difference to obtain a target action of the first virtual character;

[0015] A control module is used to control the first virtual character to perform the target action.

[0016] An embodiment of the present application provides an electronic device, including:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the control method of the virtual character provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing a method for controlling a virtual character provided in an embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement the control method of the virtual character provided in the embodiment of the present application when executed by a processor.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] After obtaining the first state parameter of the first virtual character in the virtual scene and the second state parameter of the second virtual character interacting with the first virtual character, the parameter difference between the first state parameter and the second state parameter is determined, and an action decision is made based on the parameter difference to obtain the target action of the first virtual character. Since the parameter difference can highly condense the difference between the first virtual character and the second virtual character and reduce the dimensional space of the data, the difficulty of solving the action decision can be reduced, thereby improving the efficiency of the action decision. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 1 is a schematic diagram of the architecture of a control system 100 for a virtual character provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0025] Figure 3 It is a first flow chart of the control method of the virtual character provided in the embodiment of the present application;

[0026] Figure 4 is a second flow chart of the method for controlling a virtual character provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a virtual scene provided in an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of an invalid state shielding mechanism provided in an embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of the principle of determining the cover information difference provided by an embodiment of the present application;

[0030] Figure 8 It is a schematic diagram of the principle of determining the exposure information difference provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0034] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0037] 1) Virtual scene: It is a scene displayed (or provided) when the application is running on a terminal device. The scene can be a simulation of the real world, a semi-simulation and semi-fictional virtual environment, or a purely fictional virtual environment. The virtual scene can be any of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene. The embodiment of the present application does not limit the dimension of the virtual scene. For example, the virtual scene may include the sky, land, ocean, etc. The land may include environmental elements such as deserts and cities, and the user can control the player character to move in the virtual scene.

[0038] 2) Virtual characters: images of various people and objects that can interact in a virtual scene, or movable objects in a virtual scene. The movable objects can be virtual characters, virtual animals, cartoon characters, etc., such as people and animals displayed in a virtual scene. Virtual characters can include player characters and non-player characters. For example, a virtual character can be a virtual image (i.e., a player character) in a virtual scene that represents a user. A virtual scene can include multiple virtual characters, each of which has its own shape and volume in the virtual scene and occupies a part of the space in the virtual scene.

[0039] 3) Game Agent: In a game scenario, the machine that performs reinforcement learning can be called an agent. For example, it can be an NPC driven by a neural network. The thing that interacts with it can be called the environment. The game agent makes reasonable decisions by analyzing the game environment, and can also cooperate and compete with other agents or player characters to make it behave as intelligently as humans.

[0040] 4) Reinforcement Learning (RL): It is one of the paradigms of machine learning, also known as enhanced learning. Machine learning studies how game agents make decisions about the environment in a series of decision sequences to obtain rewards (or rewards), and continuously improve their own decisions to obtain the maximum reward value. After repeated iterations, game agents can obtain an optimal mapping from perception to decision-making.

[0041] 5) Auxiliary rewards: In the reinforcement learning algorithm, in order to enable the game agent to achieve various differentiated performances and more differentiated goals, rewards are designed to assist in achieving the main goal.

[0042] 6) Self-Attention: This is a mechanism widely used in deep learning models, and is particularly effective when processing sequence data. It allows each element of the input sequence to be compared with other elements in the sequence to calculate the representation of each sequence. This mechanism enables the model to focus on the relationship between different positions in the input sequence, thereby capturing complex dependencies within the sequence.

[0043] 7) Cross-Attention: It is a mechanism that establishes associations between two different input sequences and calculates attention weights. Unlike the self-attention mechanism, the cross-attention mechanism focuses on the interaction between two different input sequences rather than the relationship between elements within a single input sequence.

[0044] 8) Convolutional Neural Networks (CNN): Convolutional networks are feedforward neural networks in which a single neuron responds only to stimuli within a limited area, and the perception areas of different neurons overlap, thus covering the entire field of view. Its artificial neurons can respond to surrounding units within a part of the coverage area, and have excellent performance in large image processing.

[0045] 9) Behavior Tree: A tree structure used to control the behavior decisions of NPCs in the game.

[0046] 10) Proximal Policy Optimization (PPO): It is an online deep reinforcement learning algorithm based on policy gradient optimization for continuous or discrete action spaces.

[0047] 11) Cloud gaming: Also known as gaming on demand, it means deploying a game program on a server and running an instance of the game program (referred to as a game instance). The game instance sends the game data output during the running process to the browser page of the user terminal. The page calls the media component of the browser to decode the game data and renders the real-time game screen during the game according to the decoding result. When the page detects the user's operation on the game screen, it will report it to the game instance running on the server. When the game data generated by the game instance in response to the operation is received, the decoding and rendering process will be repeated, so that the game screen changes according to the user's operation are presented on the page.

[0048] In other words, cloud gaming is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud gaming scenario, the game is not played on the user terminal (such as the player's game terminal), but on the cloud server, which renders the game scene into an audio and video stream and transmits it to the user terminal over the network. In this way, the user terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input commands and send them to the cloud server.

[0049] 12) Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0050] Taking the First-Person Shooting Game (FPS) as an example, there are usually multiple NPCs in FPS games. In related technologies, the behaviors of multiple NPCs are usually constrained by formulating complex rules, thereby ensuring that multiple NPCs have coordinated anthropomorphic behaviors.

[0051] However, since the rules of the behavior tree are single and fixed, it is easy for players to see through the routine. At the same time, the production cost of the behavior tree is long and it is difficult to cope with the ever-changing new version product requirements. In other words, the solutions provided by the relevant technology are difficult to apply to shooting games with complex and diverse maps. At the same time, the solutions based on the behavior tree have fixed strategies, and it is difficult for multiple NPCs to cooperate with each other. In addition, the production cycle of the behavior tree is long, and it is difficult to cope with the ever-changing new versions.

[0052] In view of the above technical problems, the embodiment of the present application can accelerate the collection of training data for strategy learning by means of large-scale distributed reinforcement learning, and can also model the reinforcement learning system based on information difference, reduce the dimensional space, so as to solve the problem of multi-target coordination anthropomorphism. In addition, the embodiment of the present application can also use convolutional neural networks and attention mechanisms to deal with the model construction problem in multi-target battle scenes. Specifically, the embodiment of the present application provides a training framework based on large-scale reinforcement learning, and combines the proximal strategy optimization algorithm. In a complex three-dimensional game scene, the abstract problem of multi-target coordination is converted into the optimization target of machine learning by using information difference, so as to solve the problem of multi-target coordination. In addition, under multi-target confrontation training, the encoding based on information difference can effectively reduce the number of enemy and friendly states, and has the effect of reducing dimensions. Finally, for the situation where the number of both sides is large and not fixed during the battle, the embodiment of the present application can use convolutional neural networks and attention mechanisms to model the model structure, which can not only highly condense the state information, but also reduce the difficulty of training.

[0053] That is to say, the embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for controlling a virtual character, which can reduce the difficulty of making action decisions for the virtual character in the virtual scene, thereby improving the efficiency of action decisions. The electronic device provided by the embodiment of the present application is described below. The electronic device provided by the embodiment of the present application can be implemented as a terminal device (corresponding to a stand-alone game application), or implemented in collaboration with a terminal device and a server (corresponding to a networked game application). The following is an example of the control method of the virtual character provided by the embodiment of the present application implemented in collaboration with a server and a terminal device.

[0054] Before introducing the architecture of the control system of the virtual character provided in the embodiment of the present application, the game mode involved in the embodiment of the present application is first introduced. The scheme for the collaborative implementation of the terminal device and the server mainly involves two game modes, namely the local game mode and the cloud game mode, wherein the local game mode refers to the terminal device and the server jointly running the game processing logic, the operation instructions input by the player in the terminal device, part of which is processed by the terminal device running the game logic, and the other part is processed by the server running the game logic, and the game logic processing run by the server is often more complicated and requires more computing power; the cloud game mode refers to the game logic processing run entirely by the server (such as a cloud server), and the cloud server renders the game scene data into an audio and video stream, which is then transmitted to the terminal device for display via the network. In other words, the terminal device only needs to have basic streaming media playback capabilities and the ability to obtain the player's operation instructions and send them to the server.

[0055] The following describes the architecture of the control system of the virtual character provided in the embodiment of the present application.

[0056] For example, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a control system 100 for a virtual character provided in an embodiment of the present application. Figure 1 As shown, the control system 100 of the virtual character includes: a server 200 (such as a game background server), a network 300 and a terminal device 400, wherein the network 300 can be a local area network or a wide area network, or a combination of the two, the terminal device 400 is a terminal device associated with a player, and a client 410 runs on the terminal device 400. The client 410 can be various types of game clients, such as FPS game clients, third-person shooter (TPS, Third-Person Shooter) game clients, role-playing game clients, and massively multiplayer online role-playing game clients, etc.

[0057] In some embodiments, taking the client 410 as an FPS game client as an example, a virtual scene can be displayed in the human-computer interaction interface of the client 410, and a first virtual character (such as an NPC) and a second virtual character interacting with the first virtual character (such as a player character controlled by a player) can be displayed in the virtual scene. The server 200 can obtain the first state parameters of the first virtual character (such as a first distance between the first virtual character and a cover in the virtual scene, a first exposure ratio of the first virtual character exposed to the field of vision of the second virtual character, whether the first virtual character can see the second virtual character, and whether the first virtual character can hear the second virtual character, etc.) and the second state parameters of the second virtual character through the network 300. The server 200 may determine the parameter difference between the first state parameter and the second state parameter (for example, including the cover information difference, the exposure information difference, the field of view information difference, and the auditory information difference, etc.), and make an action decision based on the parameter difference through the pre-trained action decision model to obtain the target action of the first virtual character. Then, the server 200 may send the target action to the client 410, so that the client 410 controls the first virtual character to perform the target action.

[0058] In other embodiments, the control method of the virtual character provided in the embodiment of the present application can also be implemented by the terminal device alone. Figure 1Taking the terminal device 400 shown in as an example, a pre-trained action decision model is deployed locally on the terminal device 400. After obtaining the first state parameter of the first virtual character and the second state parameter of the second virtual character, the parameter difference between the first state parameter and the second state parameter can be determined based on the computing power of the terminal device 400 itself, and an action decision is made based on the parameter difference through the pre-trained action decision model deployed locally on the terminal device 400 to obtain the target action of the first virtual character. The embodiments of the present application do not make specific limitations on this.

[0059] It should be noted that the virtual scene in the control method of the virtual character provided in the embodiment of the present application can be completely based on the output of the terminal device, or based on the coordinated output of the terminal device and the server, for example, it can be completely based on Figure 1 The graphics processing hardware computing capability of the terminal device 400 shown in the figure completes the relevant data calculation and output of the virtual scene, wherein the types of graphics processing hardware include a central processing unit (CPU) and a graphics processing unit (GPU). For example, when forming a visual perception of a virtual scene, the terminal device 400 calculates the data required for display through the graphics computing hardware, and completes the loading, parsing and rendering of the display data, and the graphics output hardware outputs a video frame that can form a visual perception of the virtual scene, for example, presenting a two-dimensional video frame on the display screen of a smartphone, or projecting a video frame that realizes a three-dimensional display effect on the lenses of augmented reality / virtual reality glasses; in addition, in order to enrich the perception effect, the terminal device 400 can also use different hardware to form one or more of auditory perception, tactile perception, motion perception and taste perception.

[0060] Of course, it is also possible to rely on the computing power of the server 200 to complete the virtual scene calculation and output the virtual scene on the terminal device 400. For example, taking the formation of visual perception of the virtual scene as an example, the server 200 calculates the virtual scene-related display data (such as scene data) and sends it to the terminal device 400 through the network 300. The terminal device 400 relies on the graphics computing hardware to complete the loading, parsing and rendering of the display data, and relies on the graphics output hardware to output the virtual scene to form a visual perception. For example, a two-dimensional video frame can be presented on the display screen of a smartphone, or a video frame with a three-dimensional display effect can be projected on the lenses of augmented reality / virtual reality glasses. As for the perception of the form of the virtual scene, it can be understood that the corresponding hardware output of the terminal device 400 can be used, such as using a microphone to form auditory perception, using a vibrator to form tactile perception, and so on.

[0061] In addition, it should be noted that Figure 1The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0062] In other embodiments, the terminal device or server can also implement the control method of the virtual character provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a shooting game APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0063] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a terminal device as an example, see Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0064] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0066] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0067] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0069] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0070] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0071] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0072] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0073] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The control device 555 of the virtual character stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5551, determination module 5552, action decision module 5553, control module 5554, training module 5555 and test module 5556. These modules are logical, so they can be arbitrarily combined or further divided according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the control device 555 of the virtual character excludes the implementation of only including the acquisition module 5551, the determination module 5552, the action decision module 5553 and the control module 5554. The functions of each module will be explained below.

[0074] It should be noted that, when the electronic device 500 is a server, the above-mentioned user interface 530 (including the output device 531 and the input device 532 ), the presentation module 553 and the input processing module 554 may be default.

[0075] The control method of the virtual character provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the terminal device provided in the embodiment of the present application.

[0076] For example, see Figure 3 , Figure 3 is a first flow chart of a method for controlling a virtual character provided in an embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0077] It should be noted that Figure 3 The method shown can be executed by various forms of computer programs running on the terminal device, and is not limited to the client. For example, it can also be the operating system, software module, script, and applet mentioned above. Therefore, the example of the client in the following should not be regarded as a limitation on the embodiments of the present application. In addition, for the convenience of expression, the terminal device and the client running on the terminal device are not specifically distinguished in the following.

[0078] In step 101, a first state parameter of a first virtual character in a virtual scene and a second state parameter of a second virtual character interacting with the first virtual character are obtained.

[0079] Here, the first virtual character may be a non-player character or a player character (eg, a player character in a trusteeship state); the second virtual character may also be a non-player character or a player character.

[0080] For example, taking the first virtual character as a plurality of non-player characters in a first camp in a virtual scene (for example, as a defender) and the second virtual character as a plurality of player characters in a second camp in the virtual scene (for example, as an attacker) as an example, the state parameters of each non-player character in the first camp and the state parameters of each player character in the second camp can be obtained. For example, taking non-player character 1 among the plurality of non-player characters as an example, the state parameters of non-player character 1 may include: the distance between non-player character 1 and a cover in the virtual scene, the exposure ratio of non-player character 1 exposed to the field of view of each player character, whether non-player character 1 can see the player characters in the second camp, and whether non-player character 1 can hear the player characters in the second camp, etc.; taking player character 1 among the plurality of player characters controlled by player 1 as an example, the state parameters of player character 1 may include: the distance between player character 1 and a cover in the virtual scene, the exposure ratio of player character 1 exposed to the field of view of each non-player character, whether player character 1 can see the non-player characters in the first camp, and whether player character 1 can hear the non-player characters in the first camp, etc.

[0081] In some embodiments, when the first virtual character is a plurality of non-player characters in a first camp in a virtual scene, and the second virtual character is a plurality of player characters in a second camp in the virtual scene, the above-mentioned step 101 can be implemented in the following manner: obtaining first state parameters of non-player characters that have not been eliminated from the plurality of non-player characters included in the first camp, and obtaining second state parameters of player characters that have not been eliminated from the plurality of player characters included in the second camp.

[0082] For example, taking FPS games as an example, suppose there are 10 NPCs in the game scene (i.e., the first virtual character, as the defender in the game scene), and also suppose there are 4 player characters in the game scene (i.e., the second virtual character, as the attacker in the game scene). These 4 player characters can be controlled by 4 different real players, and their purpose is to obtain the virtual treasure chest in the game scene. As the game progresses, NPCs or player characters will be eliminated. Therefore, in the process of controlling NPCs, only the state parameters of the NPCs that are not eliminated and the state parameters of the player characters that are not eliminated can be obtained. In other words, when making action decisions for NPCs, the information of the NPCs that are eliminated and the player characters that are eliminated can be shielded (this information has little effect on the action decisions of the NPCs that are not eliminated), and only the NPCs that are not eliminated and the player characters that are not eliminated can be focused on, so that the difficulty of action decisions can be further reduced while ensuring the accuracy.

[0083] In step 102 , a parameter difference between a first state parameter and a second state parameter is determined.

[0084] In some embodiments, the first state parameter can represent the state of the first virtual character in multiple dimensions (for example, including distance dimension, exposure dimension, field of view dimension, and hearing dimension, etc.), and the second state parameter can represent the state of the second virtual character in multiple dimensions. Step 102 can be implemented in the following manner: determine the parameter difference (or information difference) corresponding to the first state parameter and the second state parameter in each dimension.

[0085] For example, the above-mentioned multiple dimensions may include a distance dimension, that is, the first state parameter may include a first distance between the first virtual character and the shelter in the virtual scene, and the second state parameter may include a second distance between the second virtual character and the shelter. Then, the above-mentioned determination of the parameter difference corresponding to the first state parameter and the second state parameter in each dimension may be achieved in the following manner: the difference between the first distance and the second distance is used as the parameter difference corresponding to the distance dimension (i.e., the shelter information difference). When the difference between the first distance and the second distance is greater than zero, it indicates that the first virtual character is farther from the shelter in the virtual scene than the second virtual character, that is, the first virtual character is at a disadvantage in terms of distance; when the difference between the first distance and the second distance is equal to zero, it indicates that the distances between the first virtual character and the second virtual character and the shelter in the virtual scene are the same, that is, the first virtual character has no advantage in terms of distance; when the difference between the first distance and the second distance is less than zero, it indicates that the first virtual character is closer to the shelter in the virtual scene than the second virtual character, that is, the first virtual character is at an advantage in terms of distance.

[0086] It should be noted that, when the first virtual character is a plurality of non-player characters in a first camp in a virtual scene (for example, m NPCs, where m is a positive integer greater than 1), and the second virtual character is a plurality of player characters in a second camp in the virtual scene (for example, n player characters, where n is a positive integer greater than 1), the above-mentioned first state parameter may include m first distances between the m NPCs and the cover in the virtual scene, and the second state parameter may include n second distances between the n player characters and the cover, and the cover information difference between each NPC and each player character may be calculated, that is, the total number of the above-mentioned cover information differences is m*n.

[0087] For example, the above-mentioned multiple dimensions may also include an exposure dimension, that is, the first state parameter may include a first exposure ratio of the first virtual character in the field of view of the second virtual character (for example, the ratio of the body part of the first virtual character exposed in the field of view of the second virtual character to the total body part), and the second state parameter may include a second exposure ratio of the second virtual character in the field of view of the first virtual character (that is, the ratio of the body part of the second virtual character exposed in the field of view of the first virtual character to the total body part). The above-mentioned determination of the parameter difference corresponding to the first state parameter and the second state parameter in each dimension may be achieved in the following manner: the difference between the first exposure ratio and the second exposure ratio is used as the parameter difference corresponding to the exposure dimension (that is, the exposure information difference). When the difference between the first exposure ratio and the second exposure ratio is less than zero, it indicates that the first virtual character exposes less body parts than the second virtual character, that is, the first virtual character is at an advantage in terms of exposure; when the difference between the first exposure ratio and the second exposure ratio is equal to zero, it indicates that the body parts exposed by the first virtual character and the body parts exposed by the second virtual character are the same, that is, the first virtual character has no advantage in terms of exposure; when the difference between the first exposure ratio and the second exposure ratio is greater than zero, it indicates that the first virtual character exposes more body parts than the second virtual character, that is, the first virtual character is at a disadvantage in terms of exposure.

[0088] It should be noted that, when the first virtual character is a plurality of non-player characters in the first camp in the virtual scene (for example, m NPCs, where m is a positive integer greater than 1), and the second virtual character is a plurality of player characters in the second camp in the virtual scene (for example, n player characters, where n is a positive integer greater than 1), the above-mentioned first state parameter may include the first exposure ratios corresponding to the m NPCs in the fields of view of the n player characters, and the second state parameter may include the second exposure ratios corresponding to the n player characters in the fields of view of the m NPCs, and then the exposure information difference between each NPC and each player character can be calculated, that is, the total number of the above-mentioned exposure information differences is m*n.

[0089] For example, the above-mentioned multiple dimensions may also include a field of view dimension, that is, the first state parameter may include a first field of view range of the first virtual character, and the second state parameter may include a second field of view range of the second virtual character. Then, the above-mentioned determination of the parameter difference corresponding to the first state parameter and the second state parameter in each dimension may be achieved in the following manner: when the second virtual character is within the first field of view range and the first virtual character is outside the second field of view range (that is, the first virtual character can see the second virtual character, but the second virtual character cannot see the first virtual character), a first set value (for example, 1) is used as the parameter difference corresponding to the field of view dimension, wherein the first set value represents that the amount of information (that is, the amount of effective information) perceived by the first virtual character at the visual level (that is, the reference information used to assist subsequent decision-making) is greater than the amount of information perceived by the second virtual character at the visual level (that is, the amount of effective information perceived by the first virtual character at the visual level is greater than that of the second virtual character, that is, the first virtual character has a field of view advantage over the second virtual character); when the second virtual character is within the first field of view range and the first virtual character is outside the second field of view range (that is, the first virtual character and the second virtual character are outside the first field of view range), the first set value (for example, 1) is used as the parameter difference corresponding to the field of view dimension. When the second virtual character is outside the first field of view and the first virtual character is outside the second field of view (i.e., the first virtual character cannot see the second virtual character, and the second virtual character cannot see the first virtual character), a second setting value (e.g., 0) may be used as the parameter difference corresponding to the field of view dimension, wherein the second setting value indicates that the amount of effective information perceived by the first virtual character at the visual level is equal to the amount of effective information perceived by the second virtual character at the visual level (i.e., the first virtual character does not have a field of view advantage over the second virtual character); when the second virtual character is outside the first field of view and the first virtual character is within the second field of view (i.e., the first virtual character cannot see the second virtual character, but the second virtual character can see the first virtual character), a third setting value (e.g., -1) may be used as the parameter difference corresponding to the field of view dimension, wherein the third setting value indicates that the amount of effective information perceived by the first virtual character at the visual level is less than the amount of effective information perceived by the second virtual character at the visual level (i.e., the second virtual character has a field of view advantage over the first virtual character, that is, at the visual level, the first virtual character is at a disadvantage).

[0090] For example, the above-mentioned multiple dimensions may also include an auditory dimension, that is, the first state parameter may include the first hearing of the first virtual character, and the second state parameter may include the hearing of the second virtual character. Then, the above-mentioned determination of the parameter difference corresponding to the first state parameter and the second state parameter in each dimension may also be achieved in the following manner: when the first virtual character can hear the second virtual character, and the second virtual character cannot hear the first virtual character, a first setting value (for example, 1) is used as the parameter difference corresponding to the auditory dimension, wherein the first setting value indicates that the amount of effective information perceived by the first virtual character at the auditory level is greater than the amount of effective information perceived by the second virtual character at the auditory level (that is, the amount of effective information perceived by the first virtual character at the auditory level is greater than that of the second virtual character, that is, the first virtual character has an auditory advantage over the second virtual character); when the first virtual character can hear the second virtual character, and the second virtual character can hear the first virtual character When the first virtual character cannot hear the second virtual character, or when the second virtual character cannot hear the first virtual character, a second setting value (for example, 0) is used as the parameter difference corresponding to the auditory dimension, wherein the second setting value represents that the amount of effective information perceived by the first virtual character at the auditory level is equal to the amount of effective information perceived by the second virtual character at the auditory level (that is, the first virtual character does not have an auditory advantage over the second virtual character); when the first virtual character cannot hear the second virtual character, and the second virtual character can hear the first virtual character, a third setting value (for example, -1) is used as the parameter difference corresponding to the auditory dimension, wherein the third setting value represents that the amount of effective information perceived by the first virtual character at the auditory level is less than the amount of effective information perceived by the second virtual character at the auditory level (that is, the second virtual character has an auditory advantage over the first virtual character, that is, at the auditory level, the first virtual character is at a disadvantage).

[0091] In step 103, an action decision is made based on the parameter difference to obtain a target action of the first virtual character.

[0092] In some embodiments, following the above, step 103 can be implemented in the following manner: weighted summation of multiple parameter differences corresponding to multiple dimensions (for example, including cover information difference, exposure information difference, field of view information difference, and auditory information difference, etc.) is performed to obtain a total parameter difference; action decisions are made based on the total parameter difference to obtain the target action of the first virtual character.

[0093] By way of example, the above-mentioned action decision based on total parameter difference can be implemented in the following ways to obtain the target action of the first virtual character: feature extraction of the total parameter difference to obtain the total parameter difference feature; based on the total parameter difference feature, action decision is made through a pre-trained action decision model to obtain the target action of the first virtual character.

[0094] In other embodiments, see Figure 4 , Figure 4 is a second flow chart of the method for controlling a virtual character provided in an embodiment of the present application. Figure 4 As shown, Figure 3 Step 103 shown may be performed by Figure 4 Steps 1031 to 1033 shown in the figure are implemented by combining Figure 4 The steps shown are explained.

[0095] In step 1031, attention processing is performed on the first state parameter and the second state parameter respectively, and a first attention feature and a second attention feature are obtained correspondingly.

[0096] In some embodiments, the above-mentioned first attention feature may be a self-attention feature, and the second attention feature may be a cross-attention feature. Step 1031 may be implemented in the following manner: performing self-attention processing on the first state parameter to obtain the self-attention feature; performing cross-attention processing on the second state parameter to obtain the cross-attention feature.

[0097] For example, taking the first virtual character as the defender in the virtual scene (for example, multiple NPCs used to guard the treasure chest in the game scene) and the second virtual character as the attacker in the virtual scene (for example, multiple player characters used to obtain the treasure chest in the game scene) as an example, after respectively obtaining the first state parameters of the defender (i.e., the state parameters of multiple NPCs) and the second state parameters of the attacker (i.e., the state parameters of multiple player characters), the self-attention mechanism can be used to extract features of the state parameters of multiple NPCs to extract the correlation features (i.e., self-attention features) within the multiple NPCs; at the same time, the cross-attention mechanism can also be used to extract features of the state parameters of multiple player characters to extract the cross-correlation features (i.e., cross-attention features) between multiple player characters and multiple NPCs. In this way, by introducing the attention mechanism, not only can the key associations within multiple NPCs be extracted, but also the cross-key associations between multiple player characters and multiple NPCs can be extracted, thereby improving the rationality of subsequent action decisions. At the same time, the introduction of the attention mechanism can also well solve the dynamically changing state information, which is very suitable for multi-target confrontation scenarios.

[0098] In step 1032, feature extraction is performed on the parameter difference to obtain parameter difference features.

[0099] In some embodiments, the parameter difference can be feature extracted by a convolutional neural network to obtain parameter difference features. For example, taking the first virtual character as 10 defenders in the game scene (e.g., 10 NPCs for guarding treasure chests) and the second virtual character as 4 attackers in the game scene (e.g., 4 player characters for obtaining treasure chests) as an example, after determining the information difference tensor between the 10 defenders and the 4 attackers (assuming that the dimension of the information difference tensor is 10×4×l, where l represents the number of dimensions of the information difference, i.e., there are a total of l different dimensions of information difference), the information difference tensor can be feature extracted by a convolutional neural network to obtain information difference features.

[0100] In step 1033, an action decision is made based on the parameter difference feature, the first attention feature, and the second attention feature to obtain a target action of the first virtual character.

[0101] In some embodiments, step 1033 can be implemented in the following manner: splicing the parameter difference feature, the first attention feature (such as the self-attention feature) and the second attention feature (such as the cross-attention feature) to obtain a spliced ​​feature; making an action decision based on the spliced ​​feature through a pre-trained action decision model to obtain the target action of the first virtual character.

[0102] For example, taking the first virtual character as the 10 defenders in the game scene (for example, 10 NPCs for guarding treasure chests) and the second virtual character as the 4 attackers in the game scene (for example, 4 player characters for obtaining treasure chests) as an example, the self-attention mechanism can be used to extract features from the information of the 10 defenders to obtain self-attention features, and the cross-attention mechanism can be used to extract features from the information of the 4 attackers to obtain cross-attention features. At the same time, a convolutional neural network can be used to extract features from the information difference tensors of the 10 defenders and the 4 attackers to obtain information difference features. Subsequently, the information difference features, self-attention features, and cross-attention features can be spliced ​​to obtain spliced ​​features, and the spliced ​​features can be input into the pre-trained action decision model so that the pre-trained action decision model outputs corresponding target actions for the 10 NPCs respectively.

[0103] In other embodiments, the action decision of the first virtual character can also be made in combination with the target virtual item (such as a virtual treasure chest) in the virtual scene, wherein the task of the first virtual character is to guard the target virtual item, and the task of the second virtual character is to obtain the target virtual item. Then the above-mentioned step 1033 can also be implemented in the following manner: obtaining the state parameters of the target virtual item in the virtual scene, and performing feature extraction on the state parameters of the target virtual item (for example, feature extraction can be performed in a fully connected manner) to obtain the state parameters of the target virtual item; obtaining the first interaction parameters of the first virtual character and the target virtual item, and the second interaction parameters of the second virtual character and the target virtual item; performing feature extraction on the first interaction parameter and the second interaction parameter respectively (for example, feature extraction can be performed through a convolutional neural network) to obtain the first interaction feature and the second interaction feature correspondingly; making an action decision based on the parameter difference feature, the state feature of the target virtual item, the first interaction feature, the second interaction feature, the first attention feature and the second attention feature to obtain the target action of the first virtual character.

[0104] For example, taking the target virtual item as a treasure chest in a game scene, the first virtual character as 10 defenders in the game scene (e.g., 10 NPCs for guarding the treasure chest), and the second virtual character as 4 attackers in the game scene (e.g., 4 player characters for obtaining the treasure chest, respectively controlled by 4 different real players) as an example, the information of all treasure chests in the game scene (e.g., the information of 4 treasure chests) can be obtained, and the information of the 4 treasure chests can be feature extracted in a fully connected manner to obtain the state features of the 4 treasure chests. In addition, the convolutional neural network can be used to extract features of the interaction information (i.e., the first interaction parameter) between the 10 defenders and the 4 treasure chests to obtain the first The interactive features, and the interactive information (i.e., the second interactive parameter) of the four attackers and the four treasure chests can be extracted using a convolutional neural network to obtain the second interactive features. Subsequently, the state features of the four treasure chests, the first interactive features, the second interactive features, the information difference features, the self-attention features obtained by extracting the information of the 10 defenders through the self-attention mechanism, and the cross-attention features obtained by extracting the information of the four attackers through the cross-attention mechanism can be spliced ​​to obtain the spliced ​​features. Finally, the spliced ​​features can be input into the pre-trained action decision model so that the pre-trained action decision model outputs the corresponding target actions for the 10 NPCs. In this way, by combining the information of the treasure chest in the game scene, the interactive information between the defender and the treasure chest, and the interactive information between the attacker and the treasure chest, the actions decided can be made more reasonable.

[0105] For example, when the above-mentioned target virtual item includes a valid state and an invalid state (when the target virtual item is in an invalid state, it will not affect the subsequent actions of the first virtual character), the above-mentioned acquisition of the state parameters of the target virtual item in the virtual scene can be achieved in the following manner: Acquire the state parameters of the target virtual item in the virtual scene that is in a valid state. For example, assuming that there are a total of 4 treasure chests in the game scene, namely treasure chest 1 to treasure chest 4, as the game progresses, treasure chest 1 becomes invalid. At this time, the information of treasure chest 1 will not affect the actions of the NPC. In the subsequent action decision process, the information of treasure chest 1 can be shielded, that is, only the information of valid treasure chests (such as treasure chest 2 to treasure chest 4) can be focused on, thereby reducing the difficulty of action decision.

[0106] For example, following the above example, when the first virtual character is a plurality of non-player characters in the first camp in the virtual scene, and the second virtual character is a plurality of player characters in the second camp in the virtual scene, the above-mentioned acquisition of the first interaction parameter between the first virtual character and the target virtual item, and the second interaction parameter between the second virtual character and the target virtual item can be realized in the following manner: for the non-player characters that have not been eliminated among the plurality of non-player characters included in the first camp, the first interaction parameter between the non-player characters that have not been eliminated and the target virtual item in a valid state is acquired; for the player characters that have not been eliminated among the plurality of player characters included in the second camp, the second interaction parameter between the player characters that have not been eliminated and the target virtual item in a valid state is acquired. That is to say, in view of the situation that the number of enemies and the number of valid treasure chests will change dynamically during the battle, the embodiment of the present application can shield the virtual characters (including NPCs and player characters) that have been eliminated and the invalid treasure chests, so that the model can shield the relevant invalid information in the subsequent action decision process, and only focus on the valid information, thereby reducing the difficulty of action decision while further improving the rationality of the action decided.

[0107] In some embodiments, following the above, the above-mentioned action decision based on parameter difference characteristics, state characteristics of the target virtual object, the first interaction characteristics, the second interaction characteristics, the first attention characteristics and the second attention characteristics can be implemented in the following way to obtain the target action of the first virtual character: the parameter difference characteristics, the state characteristics of the target virtual object, the first interaction characteristics, the second interaction characteristics, the first attention characteristics and the second attention characteristics are spliced ​​to obtain spliced ​​characteristics; based on the spliced ​​characteristics, an action decision is made through a pre-trained action decision model to obtain the target action of the first virtual character.

[0108] For example, taking the target virtual item as a treasure chest in a game scene, the first virtual character as 10 defenders in the game scene (e.g., 10 NPCs for guarding the treasure chest), and the second virtual character as 4 attackers in the game scene (e.g., 4 player characters for obtaining the treasure chest, respectively controlled by 4 different real players) as an example, the information of all treasure chests in the game scene (e.g., the information of 4 treasure chests) can be obtained, and the information of the 4 treasure chests can be feature extracted in a fully connected manner to obtain the state features of the 4 treasure chests. In addition, the convolutional neural network can be used to extract features of the interaction information (i.e., the first interaction parameter) between the 10 defenders and the 4 treasure chests to obtain the first The second interaction feature is obtained by extracting the interaction information (i.e., the second interaction parameter) of the 4 attackers and the 4 treasure chests using a convolutional neural network, and then the state features of the 4 treasure chests, the first interaction feature, the second interaction feature, the information difference feature, the self-attention feature obtained by extracting the information of the 10 defenders through the self-attention mechanism, and the cross-attention feature obtained by extracting the information of the 4 attackers through the cross-attention mechanism are spliced ​​to obtain the spliced ​​feature. Finally, the spliced ​​feature can be input into the pre-trained action decision model to enable the pre-trained action decision model to output corresponding target actions for the 10 NPCs respectively.

[0109] In some embodiments, following the above example, before making an action decision through a pre-trained action decision model based on the splicing feature, the following processing may also be performed: creating a training game in a sample virtual scene, wherein the training game may include a first sample virtual character controlled by the initialized action decision model, and a second sample virtual character controlled by a behavior tree included in the sample virtual scene; obtaining a third state parameter of the first sample virtual character, and a fourth state parameter of the second sample virtual character; determining a parameter difference between the third state parameter and the fourth state parameter (e.g., a total information difference obtained by weighted summation of multiple information differences corresponding to multiple dimensions), and updating the parameters of the initialized action decision model based on the determined parameter difference. That is, in the early stage of training, the behavior tree built into the game client may be used as an opponent to train the initialized action decision model. At the same time, during the training process, the information difference between the first sample virtual character and the second sample virtual character can be used as the optimization target of machine learning. The information difference coding can not only highly condense the state characteristics, but also transform the abstract coordination problem into an objective function that can be optimized by machine learning. The embodiment of the present application can construct an objective function based on the concept of information difference. The anthropomorphic concepts such as the movement of multiple NPCs, finding cover to fight, and coordination and cooperation can be solved based on the objective function modeled by the above information difference.

[0110] In other embodiments, following the above, the updating of the parameters of the initialized action decision model based on the parameter difference can be achieved in the following ways: obtaining the estimated target action output by the initialized action decision model for the first sample virtual character; determining the feedback data (i.e., reward) of the first sample virtual character when performing the estimated target action; based on the parameter difference and the feedback data, updating the parameters of the initialized action decision model. In other words, the embodiment of the present application can not only construct an objective function based on the concept of information difference, but also add auxiliary rewards (such as including battle victory rewards, battle failure rewards, information difference rewards, and other rewards) to the objective function to further optimize the anthropomorphism, that is, the model can be trained based on the information difference and the auxiliary rewards at the same time, so that the trained model can control the NPC to reasonably use the cover to fight, the humanized style of play, and the dispersed movement, etc.

[0111] In some embodiments, each time a round of training is completed, the trained model can be tested, and the following processing can also be performed: a performance test is performed on the action decision model that has completed the i-th round to obtain a performance test result, wherein i is an integer variable that gradually increases from 1; when the performance test result indicates that the action decision model that has completed the i-th round of training does not meet the training completion condition, the action decision model that has completed the i-th round of training is trained for the i+1th round; when the performance test result indicates that the action decision model that has completed the i-th round of training meets the training completion condition, the action decision model that has completed the i-th round of training is used as the action decision model for making action decisions (i.e., the pre-trained action decision model).

[0112] For example, after each round of training, the action decision model that has completed the current round of training can be performance tested to obtain performance test results. If the performance test results indicate that the performance of the action decision model that has completed the current round of training has reached the training completion standard (for example, the accuracy of the model reaches the accuracy threshold), the training can be stopped, and the action decision model that has completed the current round of training will be used as the final action decision model for action decision-making, that is, the model will be deployed online; if the performance test results indicate that the performance of the action decision model that has completed the current round of training has not reached the training completion standard, the model can be trained again until the performance of the model reaches the training completion standard. For example, a performance test can be performed on the action decision model that has completed the first round of training to obtain a performance test result. If the performance test result indicates that the performance of the action decision model that has completed the first round of training has reached the training completion standard, the training can be stopped and the position decision model that has completed the first round of training can be deployed to the game client or the game background server; if the performance test result indicates that the performance of the action decision model that has completed the first round of training has not reached the training completion standard, the action decision model that has completed the first round of training can be trained for the second round, and then a performance test can be performed on the action decision model that has completed the second round of training. If the performance test result indicates that the performance of the action decision model that has completed the second round of training has reached the training completion standard, the training can be stopped and the action decision model that has completed the second round of training can be deployed to the game client or the game background server. If the performance test result indicates that the performance of the action decision model that has completed the second round of training has not reached the training completion standard, the action decision model that has completed the second round of training can be trained for the third round, and so on, until the performance of the model reaches the training completion standard.

[0113] In other embodiments, continuing from the above, the above-mentioned performance test of the action decision model that has completed the i-th round of training can be implemented in the following ways to obtain the performance test result: the action decision model that has completed the i-th round of training is stored in a test model pool (or evaluation model pool); the test model pool is sampled to obtain action decision model samples, and a test game is created, wherein the test game may include a first sample virtual character controlled by the action decision model that has completed the i-th round of training, and a second sample virtual character controlled by the action decision model sample; based on the game data of the first sample virtual character and the second sample virtual character in the test game, the performance test result is determined.

[0114] For example, taking the action decision model that completes the i-th round of training as the action decision model that completes the third round of training as an example, the action decision model that completes the third round of training can be stored in the test model pool, wherein the test model pool already stores the action decision model that completes the first round of training and the action decision model that completes the second round of training, that is, the test model pool currently stores three action decision models, namely, the action decision model that completes the first round of training, the action decision model that completes the second round of training, and the action decision model that completes the third round of training. Then, the action decision models stored in the test model pool can be sampled, and a test game can be created according to the sampling results, wherein the test game includes a first sample virtual character controlled by the action decision model that completes the third round of training, and a second sample virtual character controlled by the action decision model sample sampled from the test model pool (for example, it is assumed to be the action decision model that completes the second round of training), and then the game data of the first sample virtual character and the second sample virtual character in the test game can be obtained, and based on the game data, the performance test result of the action decision model that completes the third round of training is determined. For example, the game indicators (such as anthropomorphism indicators and position dispersion indicators) can be first determined based on the game data, and then the performance test results of the action decision model that has completed the third round of training can be determined based on the game indicators. When the game indicators meet the training completion standard, a performance test result can be generated to indicate that the performance of the action decision model that has completed the third round of training has met the training completion standard; when the game indicators do not meet the training completion standard, a performance test result can be generated to indicate that the performance of the action decision model that has completed the third round of training has not met the training completion standard.

[0115] It should be noted that the anthropomorphism index refers to the ratio of specific actions (i.e. actions that conform to the operating logic of real players) output by the action decision model that has completed the third round of training to all estimated target actions output; the positioning dispersion index refers to the ratio of specific actions (i.e. actions that meet the positioning dispersion conditions) output by the action decision model that has completed the third round of training to all estimated actions output.

[0116] In some embodiments, continuing from the above, the above-mentioned (i+1) round of training for the action decision model that has completed the i-th round of training can be achieved in the following manner: storing the action decision model that has completed the i-th round of training in a model database (e.g., an opponent model pool); sampling the model database to obtain action decision model samples, and creating a new training game, wherein the new training game includes a first sample virtual character controlled by the action decision model that has completed the i-th round of training, and a second sample virtual character controlled by the action decision model sample; controlling the first sample virtual character and the second sample virtual character to interact in the new training game, and based on the interaction result, performing the (i+1) round of training for the action decision model that has completed the i-th round of training.

[0117] For example, taking the action decision model that completes the i-th round of training as the action decision model that completes the fourth round of training as an example, the action decision model that completes the fourth round of training can be stored in the opponent model pool, wherein the opponent model pool already has three action decision models stored, namely, the action decision model that completes the first round of training, the action decision model that completes the second round of training, and the action decision model that completes the third round of training. That is to say, the opponent model pool currently stores four action decision models, namely, the action decision model that completes the first round of training, the action decision model that completes the second round of training, the action decision model that completes the third round of training, and After the action decision model of the fourth round of training is completed, the action decision model stored in the opponent model pool can be sampled, and a new training game can be created based on the sampling results, wherein the new training game can include a first sample virtual character controlled by the action decision model that has completed the fourth round of training, and a second sample virtual character controlled by the action decision model sample (e.g., the action decision model that has completed the third round of training) sampled from the opponent model pool. That is, the action decision model that has completed the third round of training can be used as an opponent, and the action decision model that has completed the fourth round of training can be trained for the fifth round in the created new training game. For example, the first sample virtual character and the second sample virtual character can be controlled to interact in the new training game, and then the interaction data of the first sample virtual character and the second sample virtual character in the new training game can be obtained, and based on the interaction data, the action decision model that has completed the fourth round of training can be trained for the fifth round. In other words, the attacking and defending sides can be controlled to perform self-game iterative training in the zero-sum game scenario.

[0118] In step 104, the first virtual character is controlled to perform a target action.

[0119] In some embodiments, the control process for the first virtual character can be divided into multiple control cycles, wherein each control cycle can be a set duration (e.g., 0.5 seconds), and then the first state parameter of the first virtual character at the beginning of the first control cycle and the second state parameter of the second virtual character at the beginning of the first control cycle can be obtained, and the parameter difference between the first state parameter and the second state parameter can be determined, and then the action decision can be made based on the parameter difference through the pre-trained action decision model to obtain the target action of the first virtual character in the second control cycle, wherein the second control cycle is a control cycle located after the first control cycle, and finally when the second control cycle begins, the first virtual character is controlled to perform the target action. In this way, by dividing the control process into multiple control cycles, more precise control of the first virtual character can be achieved.

[0120] Below, taking FPS games as an example, an exemplary application of the embodiment of the present application in an actual application scenario is explained.

[0121] For FPS games, the solutions provided by related technologies are usually to constrain the coordinated behavior between multiple targets (such as multiple NPCs) by formulating complex rules, so as to ensure that multiple NPCs can have coordinated anthropomorphic behaviors. However, when the rules are fixed, it is easy for players to see through the routines, and the time cost of making behavior trees is high, making it difficult to cope with the ever-changing needs of new versions of products.

[0122] In view of this, the embodiment of the present application provides a control method for a virtual character, proposes a training framework based on large-scale reinforcement learning, and combines the proximal policy optimization algorithm (PPO). In a complex three-dimensional scene, the information difference is used to abstract the multi-target coordination problem into the optimization goal of machine learning, so as to solve the problem of multi-target coordination. In addition, under multi-target adversarial training, the state encoding based on information difference can effectively reduce the number of enemy and friendly states, and has the effect of reducing the dimension. Finally, for the situation where the number of people on both sides is large and not fixed during the battle, the convolutional neural network and the attention mechanism can be used to construct the model structure, which can not only highly condense the state information, but also reduce the difficulty of training.

[0123] The following is a detailed description of the control method of the virtual character provided in the embodiment of the present application.

[0124] The control method of the virtual character provided in the embodiment of the present application can be applied to the multi-unit (e.g., multiple NPC) joint combat system of all shooting games, and can allow any number of NPCs to work closely together according to the battlefield situation, coordinate tactics with each other, and jointly achieve the goal of fighting against the player character, thereby increasing the confrontation and fighting enthusiasm of the players in the game. Figure 5 As shown, it is assumed that there are 10 NPCs (for example, as defenders) in a certain game scene, namely NPC501 to NPC510. At the same time, the game scene may also include a player character 511 controlled by the player (for example, as an attacker). These 10 NPCs will guard four treasure chests located in different positions in the game scene, assuming that they are treasure chest 512, treasure chest 513, treasure chest 514 and treasure chest 515. Since the game scene is very complex and the floor bunkers are diverse, if the behavior tree solution is used to control these 10 NPCs, it will be shown that the NPCs are relatively dull and single, and it is easy for players to find flaws in their play and thus defeat them one by one. Therefore, the embodiment of the present application can rely on reinforcement learning to train a command AI (corresponding to the above-mentioned action decision model) to command these 10 NPCs to deploy troops and guard the four treasure chests, thereby delaying the battle duration, thereby increasing the heat of the battle and the participation of the players. The trained model needs to achieve the following effects: 1. Moderate strength, NPCs can reasonably use cover to fight, play in an anthropomorphic style, move in a dispersed manner, and will not gather in groups; 2. Tactical coordination and diverse strategies. NPCs have tactical flanking behaviors to form a front and back double-team to deal with player characters.

[0125] That is to say, the embodiment of the present application can adopt a scheme based on information difference coding + attention mechanism modeling to solve the coordination problem under multi-target confrontation, wherein information difference coding can not only highly condense state characteristics, but also convert the abstract concept of coordination into an objective function that can be optimized by machine learning. In addition, for situations where there are many enemy and friendly personnel and the number is not fixed during the battle, the embodiment of the present application can adopt the attention mechanism for modeling, so that the information of the surviving personnel can be effectively aggregated, and the information of the dead personnel can be shielded, thereby reducing the difficulty of training. Finally, the offensive and defensive sides can perform iterative training of self-game in a zero-sum game scenario, and finally converge to the Nash equilibrium strategy. At this time, the strength and anthropomorphism of the model can meet the expectations of the business team.

[0126] It should be noted that zero-sum game, also known as zero-sum game, is a concept in game theory. It means that the sum of the interests of all parties in the game is zero, or a constant, that is, if one party gains, the other party must lose. In a zero-sum game, the parties are uncooperative and belong to a non-cooperative game. If the parties in the game make all participants gain or lose under different strategy combinations, it is a non-zero-sum game. In a non-zero-sum game, there is the possibility of cooperation between the parties. In addition, self-game is a process of iteratively solving the best response (BR, Best Response). The game agent can use its previous copy (such as strategy) as an opponent to train and update the current strategy and continue to iterate. By training itself in the face of an increasingly powerful opponent, it can prevent the huge ability difference from making the strategy immovable, thereby effectively increasing the training efficiency and continuously improving the model ability. Among them, the best response is when other players choose a fixed strategy, the current player adopts a strategy that can produce the most favorable results for himself. In addition, Nash equilibrium, also known as non-cooperative game equilibrium, is an important concept in game theory. In a game process, for each player, when other players do not change their strategies, the strategies of all current players are the best responses. At this time, no player can obtain the maximum benefit by only changing his own strategy while other players' strategies remain unchanged. The current joint strategy is said to have reached Nash equilibrium. Nash equilibrium is not necessarily the optimal solution for all parties to the game. It expresses that the current game environment has reached a state of equilibrium and stability.

[0127] The information difference encoding process of the state space is explained below.

[0128] In some embodiments, in a multi-target confrontation training scenario, the number of people on both sides of the enemy and the number of treasure chests are relatively large. For example, assuming that the number of people on the friendly side (assuming that it is recorded as m) is 10, the number of enemies (assuming that it is recorded as n) is 4, and the number of treasure chests (assuming that it is recorded as u) is 4, and there is interactive information between each friendly side and each enemy, such as whether friendly side A can see enemy A, and vice versa, whether enemy A can see friendly side A, and there are many similar interactive information. So if the number of all interactive information between a friendly side and an enemy is x, then the number of interactive information of all friendly sides for one enemy is m*x. Considering that the number of enemies is n, the number of interactive information between the entire friendly side and the entire enemy is m*x*n. On the contrary, assuming that the number of all interactive information of an enemy for a friendly side is also x (that is, the interactive information is symmetrical to each other, but not necessarily equal), then the number of interactive information of all enemies for all friendly sides is n*x*m, so the total number of interactive information is 2*n*x*m. This is just the interaction information between the enemy and us. If we consider the interaction information between us and the treasure chest, and the interaction information between the enemy and the treasure chest, a large amount of interaction information will appear. The following is an example of visibility:

[0129] 1. Enemy A can see your A, and your A can also see enemy A

[0130] 2. Enemy A cannot see your A, and your A cannot see enemy A.

[0131] In the technical solution provided in the embodiment of the present application, the above two situations can be combined into one situation, that is, there is no advantage between party A and enemy A. In this way, the two situations have a total of four interactive information, which can be combined into an information encoding. For example, 0 can be used to indicate that there is no information difference between each other; if enemy A can see party A, but party A cannot see enemy A, then party A is at a disadvantage to enemy A, and the interactive information between the two can be replaced by -1, indicating that party A is at a disadvantage. Similarly, if enemy A cannot see party A, but party A can see enemy A, the interactive information between the two can be replaced by 1, indicating that party A is at an advantage. In this way, only the three numbers -1, 0, and 1 are needed to represent many situations, thereby eliminating a large amount of redundant interactive information. In other words, the feature dimension can be effectively reduced by encoding the state space through information difference, thereby accelerating the training process.

[0132] It should be noted that the above is the information difference between our own party A and enemy A in the visibility dimension. Given that the number of our own party members is m and the number of enemies is n, in the visibility dimension, the information difference between all our own party members and all enemies can be represented by an m×n vision advantage matrix. Considering that our own party can have information differences of multiple different dimensions with the enemy, for example, in addition to the vision information difference, it can also include position information difference, proximity to cover information difference, etc. Assuming that there are a total of l different dimensions of information differences, then all information differences between our own party and the enemy are three-dimensional tensors of m×n×l. This approach can effectively handle the situation where there are a large number of enemies and friends and a large amount of interactive information. It can highly condense the encoding state space, reduce the information dimension, and thus accelerate the training process. Then, a convolutional neural network can be used to process this three-dimensional tensor information and adaptively extract the intrinsic associations between different members of our own party and the enemy.

[0133] The following will continue to explain the process of modeling based on the attention mechanism.

[0134] In some embodiments, in a multi-player and multi-target confrontation scenario, the number of enemies and friends, and the number of valid treasure chests will change dynamically during the battle, and the model structure needs to adapt to this dynamic change and only capture the information of surviving game characters (including player characters and NPCs) and valid treasure chests. To this end, the embodiment of the present application can shield the eliminated game characters and invalid treasure chests, and write this shielding into the above-mentioned information difference state encoding, so that the model can shield the relevant invalid states during the training process and only aggregate valid information, thereby handling the situation where the number of surviving game characters and valid treasure chests changes dynamically.

[0135] For example, see Figure 6 , Figure 6 Schematic diagram of the invalid state shielding mechanism provided by the embodiment of the present application. Figure 6 As shown in the figure, assuming that for the interactive information of 10 defenders and 4 treasure chests, if one treasure chest becomes invalid because it is touched, then the interactive information of all dimensions corresponding to the treasure chest will be blocked. Similarly, for the m*n*l-dimensional interactive information between m friendly parties and n enemy parties, if a character is eliminated, then all the information of all dimensions corresponding to the eliminated character can be blocked, that is, the neural network will not process the state information of this dimension. During the battle, the number of personnel on both sides of the attack and defense and the number of valid treasure chests will change dynamically, and the model only needs to pay attention to the information of the surviving game characters and valid treasure chests, so the above invalid state shielding mechanism can be adopted.

[0136] In addition, for the feature extraction of effective state information after shielding, the embodiment of the present application can be implemented by using an attention mechanism. For example, when training the attacker, the self-side is from the perspective of the attacker, and the enemy is the defender at this time; and when training the defender, the self-side is from the perspective of the defender, and the enemy is the attacker at this time. When applying the attention mechanism to construct the model structure, the self-attention mechanism can be used for the self-side information to automatically extract the internal correlation features of the effective members of the self-side, and the cross-attention mechanism can be used for the enemy information, that is, based on the self-side field of view, the query of the self-side gives the enemy's key (Key) a score, and extracts the value (Value) of the effective member of the enemy according to the score, so that the correlation features can be extracted between two different categories of sequence information. The introduction of the attention mechanism allows the model to extract the key correlations within the surviving members of the self-side, and can also automatically extract the cross-key correlations between the surviving members of the enemy and the surviving members of the self-side.

[0137] In other words, the above modeling method can well solve the dynamically changing state information, which is very suitable for multi-target adversarial training scenarios. In addition, for different types of information in this scenario, the following network processing can be performed:

[0138] 1) Information about the 4 treasure chests: All connected

[0139] 2) If you are training the attacker:

[0140] a) Information of 4 attackers (4×52): Self-attention mechanism

[0141] b) Information of 10 defenders (10×52): Cross-attention mechanism

[0142] 3) If training the defender:

[0143] a) Information of 4 attackers (4×52): Cross-attention mechanism

[0144] b) Information of 10 defenders (10×52): Self-attention mechanism

[0145] 4) Interaction information between 4 attackers and 4 treasure chests (4×4×6): Convolutional network

[0146] 5) Interaction information between 10 defenders and 4 treasure chests (4×10×6): Convolutional network

[0147] 6) Information difference tensor of 10 defenders and 4 attackers (10×4×l): Convolutional network

[0148] It should be noted that in practical applications, you can also directly use all the interaction information, and then use a fully connected layer instead of the attention mechanism, which can also train the corresponding effect.

[0149] The target optimization solution process is described below.

[0150] In some embodiments, the anthropomorphism of multiple targets in FPS games has always been a long-standing problem, requiring tactical coordination and strategic collaboration between multiple units, that is, how to convert the abstract concept of anthropomorphism into a specific and solvable optimization goal of reinforcement learning, which is a problem that needs to be seriously considered. In response to this problem, the embodiment of the present application can construct an objective function based on the concept of information difference. In addition, in order to simplify the learning difficulty, the embodiment of the present application can further classify the learning process of information difference (for example, including cover information difference, exposure information difference, field of view information difference, auditory information difference, etc.), so that the model can learn more detailed body movements. The following specifically describes the cover information difference, exposure information difference, field of view information difference, and auditory information difference.

[0151] For example, see Figure 7 , Figure 7 is a schematic diagram of the principle of determining the cover information difference provided by the embodiment of the present application, such as Figure 7 As shown, the distance between the attacker and the bunker can be calculated separately (assuming it is recorded as D oppo2cover ), and the distance between the defender and the bunker (assuming it is recorded as D self2cover ), then the difference between these two distances can be used as the cover information difference (assuming it is recorded as I cover (t)), that is:

[0152] I cover (t) = D oppo2cover -D self2cover

[0153] Among them, D oppo2cover Indicates the distance between the attacker and the bunker, D self2cover Indicates the distance between the defender and the cover.

[0154] That is to say, in the optimization goal, the embodiment of the present application can introduce the concept of cover information difference to estimate the distance between the own side and the enemy and the cover. Whoever is closer to the cover has a greater advantage.

[0155] For example, see Figure 8 , Figure 8 is a schematic diagram of the principle of determining the exposure information difference provided in the embodiment of the present application, such as Figure 8 As shown, the proportion of the attacker's body parts exposed to the defender's field of view to the total body parts can be calculated separately (assuming that it is recorded as N oppo_exposed ), and the proportion of the defender's body exposed to the attacker's field of vision to the total body (assuming it is recorded as N self_exposed ), then the difference between these two ratios can be used as the exposure information difference (assuming it is recorded as Ivisual (t)), that is:

[0156] I visual (t) = N oppo_exposed -N self_exposed

[0157] Among them, N oppo_exposed Indicates the proportion of the attacker's exposed body parts to the total body parts, N self_exposed Indicates the proportion of the defender's exposed body parts to the total body parts.

[0158] For example, for the visual field information difference (assuming it is recorded as I status (t)) can be represented by 1, 0, and -1, where 1 means that the player can see the enemy, but the enemy cannot see the player; 0 means that both sides can see each other or neither can see each other; -1 means that the player cannot see the enemy, but the enemy can see the player, that is:

[0159]

[0160] Among them, 1 means that the player can see the enemy, but the enemy cannot see the player; 0 means that both sides can see each other or neither can see each other; -1 means that the player cannot see the enemy, but the enemy can see the player.

[0161] For example, for the auditory information difference (assuming it is recorded as I sound (t)), can also be represented by 1, 0, -1, where 1 means that the player can hear the enemy, but the enemy cannot hear the player; 0 means that both sides can hear each other or neither can hear each other; -1 means that the player cannot hear the enemy, but the enemy can hear the player, that is:

[0162]

[0163] Among them, 1 means that the player can hear the enemy, but the enemy cannot hear the player; 0 means that both sides can hear each other or neither can hear each other; -1 means that the player cannot hear the enemy, but the enemy can hear the player.

[0164] In some embodiments, after determining the cover information difference, exposure information difference, visual field information difference, and auditory information difference, these four different types of information differences can be weighted and summed to obtain a total information difference (assuming that it is recorded as I total (t)), that is:

[0165] I total (t) = w sound ×I sound (t)+w status ×I status (t)+wvisual ×I visual (t)+w cover ×I cover (t)

[0166] Among them, w sound 、w status 、w visual and w cover They represent the weights corresponding to the auditory information difference, visual information difference, exposure information difference and cover information difference respectively.

[0167] Then, we can use the gradient descent algorithm to optimize the following objectives to solve the anthropomorphism problem:

[0168]

[0169] Among them, θ new represents the updated model parameters, θ old represents the model parameters before updating, represents the derivative of θ, E represents the expectation, s~ρ represents the state distribution, a~π(·|s) represents the policy distribution π, N represents the number of samples used each time the model parameters are updated, t represents each time step (corresponding to the control cycle mentioned above), i represents each sample trajectory, and there may be multiple time steps in each sample trajectory, a i,t represents the action at the tth time step in the i-th sample trajectory, s i,t Represents the state of the tth time step in the i-th sample trajectory.

[0170] For the anthropomorphic concepts such as multi-target movement, cover-seeking combat, and coordination in FPS games, they can all be solved based on the objective function modeled based on the above information difference, and the weights of each sub-goal can also be manually adjusted or adaptively learned according to the style requirements. In addition, the embodiment of the present application can also design auxiliary rewards as shown in Table 1 to further optimize the anthropomorphism.

[0171] Table 1 Rewards table

[0172]

[0173] In other embodiments, since the game scenes of FPS games are similar to the three-dimensional scenes in the real world, the map range is wide enough, and the scene terrain is complex and changeable enough, in order to train the performance that can be like the anthropomorphism and diversified playing styles of humans in the real world, it is necessary to build a two-player zero-sum game framework, and continuously iterate through the self-game technology to finally converge to the Nash equilibrium strategy. The specific training process is as follows:

[0174] (1) Use the behavior tree built into the game client as the opponent to train the model for round 0; (2) Store the model that has completed round 0 training in the "opponent model pool"; (3) Add the model that has completed round 0 training to the "evaluation model pool" for evaluation; (4) Sample past models from the "opponent model pool" according to certain sampling rules as opponents, and train the K-th round model to the best response; (5) Store the model that has completed round K training in the "opponent model pool"; (6) Add the model that has completed round K training to the "evaluation model pool" for evaluation; (7) Return to (4) to train the K+1-th round model; (8) Converge to the Nash equilibrium strategy; (9) Stop training iterations.

[0175] In summary, the control method of the virtual character provided in the embodiment of the present application has the following beneficial effects:

[0176] The embodiment of the present application provides a set of reinforcement learning systems based on information difference modeling, which converts the abstract problem of multi-target coordination into the optimization goal of machine learning, so as to solve the problem of multi-target coordination in FPS games. In addition, in multi-target multi-unit confrontation scenarios, state encoding based on information difference can greatly reduce the number of interactions, highly condense state information, and has the effect of reducing dimensions. Finally, in the face of dynamic changes in personnel during multi-player battles, the model structure is modeled using convolutional networks and attention mechanisms, which can adaptively extract effective internal and mutual connections, speed up the training process, and provide new solutions to the anthropomorphism problem of multiple targets in FPS games.

[0177] The following is a description of an exemplary structure of a virtual character control device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the control device 555 of the virtual character in the memory 550 may include: an acquisition module 5551 , a determination module 5552 , an action decision module 5553 and a control module 5554 .

[0178] An acquisition module 5551 is used to acquire a first state parameter of a first virtual character in a virtual scene, and a second state parameter of a second virtual character interacting with the first virtual character; a determination module 5552 is used to determine a parameter difference between the first state parameter and the second state parameter; an action decision module 5553 is used to make an action decision based on the parameter difference to obtain a target action of the first virtual character; and a control module 5554 is used to control the first virtual character to perform the target action.

[0179] In some embodiments, the action decision module 5553 is also used to perform attention processing on the first state parameter and the second state parameter respectively, and obtain the first attention feature and the second attention feature accordingly; perform feature extraction on the parameter difference to obtain the parameter difference feature; make an action decision based on the parameter difference feature, the first attention feature and the second attention feature to obtain the target action of the first virtual character.

[0180] In some embodiments, the action decision module 5553 is also used to perform self-attention processing on the first state parameter to obtain a self-attention feature, wherein the self-attention feature is used as the first attention feature; and to perform cross-attention processing on the second state parameter to obtain a cross-attention feature, wherein the cross-attention feature is used as the second attention feature.

[0181] In some embodiments, the acquisition module 5551 is further used to acquire the state parameters of the target virtual item in the virtual scene, and perform feature extraction on the state parameters of the target virtual item to obtain the state characteristics of the target virtual item; obtain the first interaction parameter between the first virtual character and the target virtual item, and the second interaction parameter between the second virtual character and the target virtual item; the action decision module 5553 is further used to perform feature extraction on the first interaction parameter and the second interaction parameter, respectively, to obtain the first interaction characteristic and the second interaction characteristic accordingly; make an action decision based on the parameter difference characteristic, the state characteristic of the target virtual item, the first interaction characteristic, the second interaction characteristic, the first attention characteristic and the second attention characteristic to obtain the target action of the first virtual character.

[0182] In some embodiments, the first virtual character includes multiple non-player characters in a first camp, and the second virtual character includes multiple player characters in a second camp; the acquisition module 5551 is also used to obtain first state parameters of non-player characters that have not been eliminated from the multiple non-player characters, and obtain second state parameters of player characters that have not been eliminated from the multiple player characters; obtain state parameters of target virtual items in a valid state in the virtual scene; for non-player characters that have not been eliminated from the multiple non-player characters, obtain first interaction parameters between the non-player characters that have not been eliminated and the target virtual items in a valid state; for player characters that have not been eliminated from the multiple player characters, obtain second interaction parameters between the player characters that have not been eliminated and the target virtual items in a valid state.

[0183] In some embodiments, the action decision module 5553 is also used to splice the parameter difference feature, the state feature of the target virtual object, the first interaction feature, the second interaction feature, the first attention feature and the second attention feature to obtain a spliced ​​feature; based on the spliced ​​feature, an action decision is made through a pre-trained action decision model to obtain the target action of the first virtual character.

[0184] In some embodiments, the control device 555 of the virtual character also includes a training module 5555, which is used to perform the following processing before the action decision module 5553 makes an action decision based on the splicing features through a pre-trained action decision model: creating a training game in a sample virtual scene, wherein the training game includes a first sample virtual character controlled by an initialized action decision model, and a second sample virtual character controlled by a behavior tree included in the sample virtual scene; obtaining a third state parameter of the first sample virtual character and a fourth state parameter of the second sample virtual character; determining a parameter difference between the third state parameter and the fourth state parameter, and updating the parameters of the initialized action decision model based on the parameter difference.

[0185] In some embodiments, the training module 5555 is also used to obtain the estimated target action output by the initialized action decision model for the first sample virtual character; determine the feedback data of the first sample virtual character when performing the estimated target action; and update the parameters of the initialized action decision model based on the parameter difference and the feedback data.

[0186] In some embodiments, the control device 555 of the virtual character also includes a testing module 5556, which is used to perform a performance test on the action decision model that has completed the i-th round of training to obtain a performance test result, wherein i is an integer variable that gradually increases from 1; the training module 5555 is also used to perform an i+1-th round of training on the action decision model that has completed the i-th round of training when the performance test result indicates that the action decision model that has completed the i-th round of training does not meet the training completion condition; the determination module 5552 is also used to use the action decision model that has completed the i-th round of training as the action decision model for making action decisions when the performance test result indicates that the action decision model that has completed the i-th round of training meets the training completion condition.

[0187] In some embodiments, the test module 5556 is further used to store the action decision model that has completed the i-th round of training into a test model pool; sample the test model pool to obtain an action decision model sample, and create a test game, wherein the test game includes a first sample virtual character controlled by the action decision model that has completed the i-th round of training, and a second sample virtual character controlled by the action decision model sample; determine the performance test result based on the game data of the first sample virtual character and the second sample virtual character in the test game.

[0188] In some embodiments, the training module 5555 is further used to store the action decision model that has completed the i-th round of training into a model database; sample the model database to obtain action decision model samples, and create a new training game, wherein the new training game includes a first sample virtual character controlled by the action decision model that has completed the i-th round of training, and a second sample virtual character controlled by the action decision model sample; control the first sample virtual character and the second sample virtual character to interact in the new training game, and based on the interaction results, perform the i+1th round of training on the action decision model that has completed the i-th round of training.

[0189] In some embodiments, the first state parameter represents the state of the first virtual character in multiple dimensions, and the second state parameter represents the state of the second virtual character in multiple dimensions; the determination module 5552 is also used to determine the parameter difference corresponding to the first state parameter and the second state parameter in each dimension; the action decision module 5553 is also used to perform weighted summation of multiple parameter differences corresponding to multiple dimensions to obtain a total parameter difference; and an action decision is made based on the total parameter difference to obtain a target action of the first virtual character.

[0190] In some embodiments, the first state parameter includes a first distance between the first virtual character and a cover in the virtual scene, and the second state parameter includes a second distance between the second virtual character and the cover; the determination module 5552 is also used to use the difference between the first distance and the second distance as the parameter difference corresponding to the distance dimension.

[0191] In some embodiments, the first state parameter includes a first exposure ratio of the first virtual character in the field of view of the second virtual character, and the second state parameter includes a second exposure ratio of the second virtual character in the field of view of the first virtual character; the determination module 5552 is also used to use the difference between the first exposure ratio and the second exposure ratio as the parameter difference corresponding to the exposure dimension.

[0192] In some embodiments, the first state parameter includes a first visual range of the first virtual character, and the second state parameter includes a second visual range of the second virtual character; the determination module 5552 is further used to use a first setting value as a parameter difference corresponding to the visual dimension when the second virtual character is within the first visual range and the first virtual character is outside the second visual range, wherein the first setting value represents that the amount of effective information perceived by the first virtual character at the visual level is greater than the amount of effective information perceived by the second virtual character at the auditory level; when the second virtual character is within the first visual range and the first virtual character is within the second visual range, or when the second virtual character is outside the first visual range and the first virtual character is outside the second visual range, use a second setting value as a parameter difference corresponding to the visual dimension, wherein the second setting value represents that the amount of effective information perceived by the first virtual character at the visual level is equal to the amount of effective information perceived by the second virtual character at the visual level; when the second virtual character is outside the first visual range and the first virtual character is within the second visual range, use a third setting value as a parameter difference corresponding to the visual dimension, wherein the third setting value represents that the amount of effective information perceived by the first virtual character at the visual level is less than the amount of effective information perceived by the second virtual character at the visual level.

[0193] In some embodiments, the first state parameter includes the first hearing of the first virtual character, and the second state parameter includes the second hearing of the second virtual character; the determination module 5552 is further used to use the first setting value as the parameter difference corresponding to the auditory dimension when the first virtual character can hear the second virtual character and the second virtual character cannot hear the first virtual character, wherein the first setting value represents that the amount of effective information perceived by the first virtual character at the auditory level is greater than the amount of effective information perceived by the second virtual character at the auditory level; when the first virtual character can hear the second virtual character and the second virtual character can hear the first virtual character, or when the first virtual character When the second virtual character cannot be heard and the second virtual character cannot hear the first virtual character, the second set value is used as the parameter difference corresponding to the auditory dimension, wherein the second set value represents that the amount of effective information perceived by the first virtual character at the auditory level is equal to the amount of effective information perceived by the second virtual character at the auditory level; when the first virtual character cannot hear the second virtual character and the second virtual character can hear the first virtual character, the third set value is used as the parameter difference corresponding to the auditory dimension, wherein the third set value represents that the amount of effective information perceived by the first virtual character at the auditory level is less than the amount of effective information perceived by the second virtual character at the auditory level.

[0194] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 ,or Figure 4 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0195] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the computer device executes the control method of the virtual character described in the embodiment of the present application.

[0196] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the control method of the virtual character provided by the present application embodiment, for example, Figure 3 ,or Figure 4 A method for controlling a virtual character is shown.

[0197] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0198] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0199] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0200] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for controlling a virtual character, characterized in that: The method comprises: Acquire a first state parameter of a first virtual character in a virtual scene, and a second state parameter of a second virtual character interacting with the first virtual character; determining a parameter difference between the first state parameter and the second state parameter; Performing an action decision based on the parameter difference to obtain a target action of the first virtual character; Control the first virtual character to perform the target action.

2. The method according to claim 1, characterized in that The performing action decision based on the parameter difference to obtain the target action of the first virtual character includes: Performing attention processing on the first state parameter and the second state parameter respectively, and obtaining a first attention feature and a second attention feature correspondingly; Performing feature extraction on the parameter difference to obtain parameter difference features; An action decision is made based on the parameter difference feature, the first attention feature, and the second attention feature to obtain a target action of the first virtual character.

3. The method according to claim 2, characterized in that The performing attention processing on the first state parameter and the second state parameter respectively to obtain a first attention feature and a second attention feature accordingly comprises: Performing self-attention processing on the first state parameter to obtain a self-attention feature, wherein the self-attention feature is used as a first attention feature; The second state parameter is subjected to cross-attention processing to obtain a cross-attention feature, wherein the cross-attention feature is used as a second attention feature.

4. The method according to claim 2 or 3, characterized in that: The performing action decision based on the parameter difference feature, the first attention feature, and the second attention feature to obtain a target action of the first virtual character includes: Acquiring state parameters of a target virtual object in the virtual scene, and performing feature extraction on the state parameters of the target virtual object to obtain state features of the target virtual object; Acquire a first interaction parameter between the first virtual character and the target virtual item, and a second interaction parameter between the second virtual character and the target virtual item; Performing feature extraction on the first interaction parameter and the second interaction parameter respectively, and obtaining a first interaction feature and a second interaction feature correspondingly; An action decision is made based on the parameter difference feature, the state feature of the target virtual object, the first interaction feature, the second interaction feature, the first attention feature, and the second attention feature to obtain a target action of the first virtual character.

5. The method according to claim 4, characterized in that The first virtual characters include a plurality of non-player characters in a first camp, and the second virtual characters include a plurality of player characters in a second camp; The step of obtaining a first state parameter of a first virtual character in a virtual scene and a second state parameter of a second virtual character interacting with the first virtual character comprises: Acquire a first state parameter of a non-player character that has not been eliminated from the plurality of non-player characters, and acquire a second state parameter of a player character that has not been eliminated from the plurality of player characters; The step of obtaining the state parameters of the target virtual object in the virtual scene includes: Acquire state parameters of a target virtual item in a valid state in the virtual scene; The obtaining of a first interaction parameter between the first virtual character and the target virtual item, and a second interaction parameter between the second virtual character and the target virtual item, comprises: For the non-player characters that have not been eliminated from the game among the plurality of non-player characters, obtaining a first interaction parameter between the non-player characters that have not been eliminated and the target virtual item that is in a valid state; For the remaining player characters among the multiple player characters, a second interaction parameter between the remaining player characters and the target virtual item in a valid state is obtained.

6. The method according to claim 4, characterized in that The performing action decision based on the parameter difference feature, the state feature of the target virtual item, the first interaction feature, the second interaction feature, the first attention feature, and the second attention feature to obtain the target action of the first virtual character includes: splicing the parameter difference feature, the state feature of the target virtual item, the first interaction feature, the second interaction feature, the first attention feature, and the second attention feature to obtain a spliced ​​feature; Based on the splicing features, an action decision is performed through a pre-trained action decision model to obtain a target action of the first virtual character.

7. The method according to claim 6, characterized in that Before performing an action decision based on the splicing feature through a pre-trained action decision model, the method further includes: Creating a training game in a sample virtual scene, wherein the training game includes a first sample virtual character controlled by an initialized action decision model, and a second sample virtual character controlled by a behavior tree included in the sample virtual scene; Acquire a third state parameter of the first sample virtual character and a fourth state parameter of the second sample virtual character; A parameter difference between the third state parameter and the fourth state parameter is determined, and parameters of the initialized action decision model are updated based on the parameter difference.

8. The method according to claim 7, characterized in that The updating of the parameters of the initialized action decision model based on the parameter difference includes: Obtaining an estimated target action output by the initialized action decision model for the first sample virtual character; Determining feedback data of the first sample virtual character when performing the estimated target action; Based on the parameter difference and the feedback data, the parameters of the initialized action decision model are updated.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: Performing a performance test on the action decision model that has completed the i-th round of training to obtain a performance test result, wherein i is an integer variable that gradually increases from 1; When the performance test result indicates that the action decision model that has completed the i-th round of training does not meet the training completion condition, performing an i+1-th round of training on the action decision model that has completed the i-th round of training; When the performance test result indicates that the action decision model that has completed the i-th round of training meets the training completion condition, the action decision model that has completed the i-th round of training is used as the action decision model for performing the action decision.

10. The method according to claim 9, characterized in that The performing of a performance test on the action decision model that has completed the i-th round of training to obtain a performance test result includes: The action decision model that has completed the i-th round of training is stored in the test model pool; Sampling the test model pool to obtain action decision model samples, and creating a test game, wherein the test game includes the first sample virtual character controlled by the action decision model that has completed the i-th round of training, and the second sample virtual character controlled by the action decision model sample; The performance test result is determined based on the game data of the first sample virtual character and the second sample virtual character in the test game.

11. The method according to claim 9, characterized in that The step of performing the (i+1) round of training on the action decision model that has completed the i-th round of training includes: The action decision model that has completed the i-th round of training is stored in the model database; Sampling the model database to obtain action decision model samples, and creating a new training game, wherein the new training game includes the first sample virtual character controlled by the action decision model that has completed the i-th round of training, and the second sample virtual character controlled by the action decision model sample; The first sample virtual character and the second sample virtual character are controlled to interact in the new training game, and based on the interaction result, the action decision model that has completed the i-th round of training is trained for the i+1th round.

12. The method according to any one of claims 1 to 11, characterized in that: The first state parameter represents the state of the first virtual character in multiple dimensions, and the second state parameter represents the state of the second virtual character in the multiple dimensions; The determining a parameter difference between the first state parameter and the second state parameter comprises: Determine a parameter difference between the first state parameter and the second state parameter corresponding to each dimension; The performing action decision based on the parameter difference to obtain the target action of the first virtual character includes: Performing weighted summation on the plurality of parameter differences corresponding to the plurality of dimensions respectively to obtain a total parameter difference; An action decision is made based on the total parameter difference to obtain a target action of the first virtual character.

13. The method according to claim 12, characterized in that The first state parameter includes a first distance between the first virtual character and a cover in the virtual scene, and the second state parameter includes a second distance between the second virtual character and the cover; The determining of the parameter difference corresponding to each of the first state parameter and the second state parameter in each dimension includes: The difference between the first distance and the second distance is used as the parameter difference corresponding to the distance dimension.

14. The method according to claim 12, characterized in that The first state parameter includes a first exposure ratio of the first virtual character in the field of view of the second virtual character, and the second state parameter includes a second exposure ratio of the second virtual character in the field of view of the first virtual character; The determining of the parameter difference corresponding to each of the first state parameter and the second state parameter in each dimension includes: The difference between the first exposure ratio and the second exposure ratio is used as the parameter difference corresponding to the exposure dimension.

15. The method according to claim 12, characterized in that The first state parameter includes a first field of view of the first virtual character, and the second state parameter includes a second field of view of the second virtual character; The determining of the parameter difference corresponding to each of the first state parameter and the second state parameter in each dimension includes: When the second virtual character is within the first visual field and the first virtual character is outside the second visual field, a first setting value is used as a parameter difference corresponding to the visual field dimension, wherein the first setting value indicates that the amount of effective information perceived by the first virtual character at the visual level is greater than the amount of effective information perceived by the second virtual character at the visual level; When the second virtual character is within the first visual field and the first virtual character is within the second visual field, or when the second virtual character is outside the first visual field and the first virtual character is outside the second visual field, a second set value is used as a parameter difference corresponding to the visual field dimension, wherein the second set value represents that the amount of effective information perceived by the first virtual character at the visual level is equal to the amount of effective information perceived by the second virtual character at the visual level; When the second virtual character is outside the first field of view and the first virtual character is within the second field of view, a third setting value is used as the parameter difference corresponding to the field of view dimension, wherein the third setting value represents that the amount of effective information perceived by the first virtual character at the visual level is less than the amount of effective information perceived by the second virtual character at the visual level.

16. The method according to claim 12, characterized in that The first state parameter includes a first hearing of the first virtual character, and the second state parameter includes a second hearing of the second virtual character; The determining of the parameter difference corresponding to each of the first state parameter and the second state parameter in each dimension includes: When the first virtual character can hear the second virtual character, and the second virtual character cannot hear the first virtual character, a first setting value is used as a parameter difference corresponding to the auditory dimension, wherein the first setting value indicates that the amount of effective information perceived by the first virtual character at the auditory level is greater than the amount of effective information perceived by the second virtual character at the auditory level; When the first virtual character can hear the second virtual character and the second virtual character can hear the first virtual character, or when the first virtual character cannot hear the second virtual character and the second virtual character cannot hear the first virtual character, a second setting value is used as the parameter difference corresponding to the auditory dimension, wherein the second setting value represents that the amount of effective information perceived by the first virtual character at the auditory level is equal to the amount of effective information perceived by the second virtual character at the auditory level; When the first virtual character cannot hear the second virtual character, and the second virtual character can hear the first virtual character, a third setting value is used as the parameter difference corresponding to the auditory dimension, wherein the third setting value represents that the amount of effective information perceived by the first virtual character at the auditory level is less than the amount of effective information perceived by the second virtual character at the auditory level.

17. A control device for a virtual character, characterized in that: The device comprises: An acquisition module, used to acquire a first state parameter of a first virtual character in a virtual scene, and a second state parameter of a second virtual character interacting with the first virtual character; a determination module, configured to determine a parameter difference between the first state parameter and the second state parameter; an action decision module, configured to make an action decision based on the parameter difference to obtain a target action of the first virtual character; A control module is used to control the first virtual character to perform the target action.

18. An electronic device, characterized in that: include: A memory for storing computer executable instructions; The processor is used to implement the control method of the virtual character described in any one of claims 1 to 16 when executing the computer executable instructions stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the control method of the virtual character according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instructions are executed by a processor, the method for controlling a virtual character according to any one of claims 1 to 16 is implemented.

Images