Action decision-making method and apparatus for virtual character, and device and storage medium

Abstract

The present application belongs to the technical field of artificial intelligence. Disclosed are an action decision-making method and apparatus for a virtual character, and a device and a storage medium. The method comprises: acquiring state information, which is used for representing a matchup state of a matchup that a first virtual character is in; inputting the state information into an action decision-making model, so as to obtain n target sub-actions which are serially outputted by n action output heads in the action decision-making model, wherein different action output heads correspond to different action types, the n action output heads are connected in series on the basis of a dependency relationship between the action types, and the dependency relationship is used for representing a dependency limitation condition between the sub-actions under different action types; and controlling the first virtual character to execute a target action formed by the n target sub-actions. By means of the embodiments provided in the present application, a dependency relationship between all action types can be taken into consideration when action decision-making is realized by means of an action decision-making model, thereby improving the rationality of the action decision-making.

Description

Virtual character action decision method, device, equipment and storage medium

[0001] This application claims priority to Chinese patent application number 202311195505.3, filed on September 15, 2023, entitled “Virtual character action decision-making method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for making action decisions for a virtual character. Background Art

[0003] Nowadays, in the process of planning video games, the behavior logic of NPCs (Non-Player Characters) is usually made as consistent as possible with that of real players so that the NPCs in the game have high anthropomorphism.

[0004] In related technologies, behavior trees are commonly used to implement logical decision-making for NPC behaviors. Behavior trees are tools for implementing complex NPC character behaviors. When controlling an NPC's actions, a computer device traverses the behavior tree starting from its root node in the order of execution until it reaches a terminal state. During the behavior tree traversal, the computer device determines the next node to execute based on the status information (success, failure, or running) returned by different leaf nodes and pre-defined rules, thereby implementing logical decision-making for the NPC's behavior.

[0005] However, the behavior tree structure is a rule-based algorithm. If the set rules remain unchanged, the NPC's performance in the scene will be too monotonous. If the NPC needs to have a high degree of humanization, complex rules need to be set in advance, resulting in high labor costs and poor adaptability to different map scenarios.

[0006] Summary of the Invention

[0007] The present invention provides a method, apparatus, device, and storage medium for making action decisions for a virtual character. The technical solution is as follows:

[0008] In one aspect, an embodiment of the present application provides a method for determining an action of a virtual character, the method being executed by a computer device and comprising:

[0009] Acquiring status information, where the status information is used to represent a game status of the first virtual character;

[0010] Inputting the state information into an action decision model to obtain n target sub-actions serially output by n action output heads in the action decision model, wherein different action output heads correspond to different action types, and the n action output heads are serially connected based on a dependency relationship between the action types, wherein the dependency relationship is used to characterize dependency restrictions between sub-actions under different action types, and n is a positive integer;

[0011] The first virtual character is controlled to execute a target action consisting of n target sub-actions.

[0012] On the other hand, this embodiment provides a device for making action decisions for a virtual character, the device comprising:

[0013] An acquisition module, configured to acquire status information, wherein the status information is used to represent a game status of the first virtual character;

[0014] a decision module, configured to input the state information into an action decision model to obtain n target sub-actions serially output by n action output heads in the action decision model, wherein different action output heads correspond to different action types, and the n action output heads are serially connected based on a dependency relationship between the action types, wherein the dependency relationship is used to characterize the dependency restrictions between sub-actions under different action types, and n is a positive integer;

[0015] A control module is used to control the first virtual character to perform a target action consisting of n target sub-actions.

[0016] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the action decision method of the virtual character as described in the above aspects.

[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer instruction is stored. The at least one computer instruction is loaded and executed by a processor to implement the action decision method of the virtual character as described in the above aspects.

[0018] In another aspect, embodiments of the present application provide a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the action decision method for a virtual character provided in the above aspects.

[0019] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0020] In an embodiment of the present application, an action decision model is used to implement action decisions based on state information, thereby determining a reasonable target action based on the current state. The action decision model includes n action output heads connected in series, each of which is used to output different target sub-actions. The n action output heads are connected in series based on the dependencies between action types. This allows the dependencies between different actions to be considered when making action decisions, and in the process of serially outputting the n target sub-actions, the n target sub-actions are causally related in time sequence, thereby improving the rationality of the target action and enhancing the anthropomorphism of the virtual character. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 shows a block diagram of a computer system according to an exemplary embodiment of the present invention;

[0022] FIG2 shows a flow chart of a method for making action decisions for a virtual character provided by an exemplary embodiment of the present application;

[0023] FIG3 shows a schematic diagram of a ray detection solution provided by an exemplary embodiment of the present application;

[0024] FIG4 shows a schematic diagram of a two-dimensional depth map provided by an exemplary embodiment of the present application;

[0025] FIG5 shows a schematic diagram of a user interface of an application program for providing a virtual environment provided by an exemplary embodiment of the present application;

[0026] FIG6 shows a flowchart of a process for determining a target action provided by an exemplary embodiment of the present application;

[0027] FIG7 shows a schematic diagram of an autoregressive structure of n action output heads provided by an exemplary embodiment of the present application;

[0028] FIG8 is a schematic diagram showing a method of performing action masking on a parallel action output head in the related art;

[0029] FIG9 is a schematic diagram showing the influence of the movement state on the movement direction and the orientation direction superimposed on each other, provided by an exemplary embodiment of the present application;

[0030] FIG10 is a schematic diagram showing visual situation perception provided by an exemplary embodiment of the present application;

[0031] FIG11 is a schematic diagram showing a method for determining an effective aiming range in a vertical direction according to an exemplary embodiment of the present application;

[0032] FIG12 is a schematic diagram showing an effective aiming range in the horizontal direction provided by an exemplary embodiment of the present application;

[0033] FIG13 shows a schematic diagram of an action decision model provided by an exemplary embodiment of the present application;

[0034] FIG14 shows a flowchart of an action decision model training process provided by an exemplary embodiment of the present application;

[0035] FIG15 is a schematic diagram showing the interaction between a client and a server for training an action decision model during a training process provided by an exemplary embodiment of the present application;

[0036] FIG16 is a schematic diagram showing a decision mode for implementing an action decision request through application and training in an exemplary embodiment of the present application;

[0037] FIG17 is a schematic diagram showing a decision mode for implementing an action decision request in an application process according to an exemplary embodiment of the present application;

[0038] FIG18 shows a schematic structural diagram of an action decision-making device for a virtual character provided by an exemplary embodiment of the present application;

[0039] FIG19 shows a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0041] First, the nouns involved in the embodiments of this application are introduced.

[0042] An intelligent agent can perceive the environment through sensors and act on it through actuators. For each possible perception sequence, the agent should select an action to perform that maximizes its performance, given the instructions and evidence provided by the perception sequence. In this embodiment, a virtual character, acting as an intelligent agent, perceives the virtual environment, acquires environmental perception information, and executes the target action determined by the action decision model.

[0043] A virtual environment is a virtual environment displayed (or provided) when an application is running on a computer device. This virtual environment can be a simulation of the real world, a semi-simulated and semi-imaginary environment, or a purely imaginary environment. This application does not limit this to any of the following: a two-dimensional virtual environment, a 2.5-dimensional virtual environment, or a three-dimensional virtual environment. The following embodiments illustrate a three-dimensional virtual environment.

[0044] A virtual character refers to an movable object in a virtual environment. The movable object can be at least one of a virtual character, a virtual animal, and an anime character. Optionally, when the virtual environment is a three-dimensional virtual environment, the virtual character can be a three-dimensional virtual model. Each virtual character has its own shape and volume in the three-dimensional virtual environment and occupies a portion of the space in the three-dimensional virtual environment. Optionally, the virtual character is a three-dimensional character constructed based on three-dimensional human skeleton technology, and the virtual character achieves different external images by wearing different skins. In some implementations, the virtual character can also be implemented using a 2.5-dimensional or 2-dimensional model, which is not limited in the embodiments of the present application.

[0045] In related technologies, the action decision-making process of a virtual character is implemented through a hierarchical behavior tree structure based on artificial rules. The search and execution starts from the root node and returns one of the execution results of "success", "failure" or "running", thereby controlling the behavior of the virtual character.

[0046] However, the rules of behavior trees are unchanging. Executing virtual character action decisions based on behavior trees results in fixed and single virtual character performance, making it difficult to achieve complex control of the virtual character's behavioral logic. Furthermore, in complex scenarios, behavior trees require a large number of rules, resulting in significant labor costs.

[0047] In addition, in related technologies, supervised learning may be used to train a network model that conforms to human player behavior by imitating the control data generated by real players controlling virtual characters.

[0048] However, due to the complexity and diversity of map scenes, not all map scenes have enough real players for computer equipment to collect data. Therefore, using supervised learning to train action decision models wastes a lot of time and manpower costs. In addition, the decision strategy of the action decision model trained by supervised learning is too fit to the distribution of training data, making it difficult to achieve effective action decision-making in actual deployment.

[0049] Therefore, an embodiment of the present application provides an action decision method for a virtual character, which obtains state information and outputs target sub-actions serially on n action output heads based on the state information according to an action decision model completed through intensive training, so that the output target sub-actions have high rationality and anthropomorphism.

[0050] The computer device in this application can be a desktop computer, a laptop computer, a mobile phone, a tablet computer, a desktop computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, an intelligent voice interaction device, a smart home appliance, and a vehicle-mounted terminal, etc. An application that supports a virtual environment, such as an application that supports a three-dimensional virtual environment, is installed and run in the computer device. The application can be any one of a virtual reality application, a three-dimensional map program, a TPS (Third-Person Shooter) game, an FPS (First-Person Shooter) game, and a MOBA (Multiplayer Online Battle Arena) game. Optionally, the application can be a stand-alone application, such as a stand-alone 3D game program, or a network-connected application. The following embodiments are illustrated by using applications in games.

[0051] Games based on virtual environments are often composed of one or more game world maps. The virtual environment in the game simulates scenes in the real world. The target virtual characters can walk, run, jump, shoot, fight, drive, climb, glide, switch to using virtual props, use virtual props to attack other virtual characters, and other actions in the virtual environment.

[0052] Please refer to Figure 1, which shows a block diagram of a computer system according to an exemplary embodiment of the present application. The computer system may include: a terminal 110, a first server 120, and a second server 130.

[0053] In the embodiment of the present application, terminal 110 includes, but is not limited to, mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals. Terminal 110 runs an application 111 that supports a virtual environment. This application can be a TPS (Third-Person Shooter) game or an FPS (First-Person Shooter) game. When terminal 110 runs application 111, the user interface of application 111 is displayed on the screen of terminal 110. A user uses terminal 110 to control a virtual character in the virtual environment to perform activities, or the terminal controls an NPC (Non-Player Character) in the virtual environment to perform activities. The activities of the virtual character include, but are not limited to, adjusting body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, throwing, and performing skills.

[0054] The first server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In an embodiment of the present application, the first server 120 is used to provide background services for applications that support a three-dimensional virtual environment and may be a DS (Dedicated Server). Optionally, the first server 120 performs primary computing tasks, and the terminal performs secondary computing tasks; alternatively, the first server 120 performs secondary computing tasks, and the terminal performs primary computing tasks; alternatively, a distributed computing architecture is employed between the first server 120 and the terminal for collaborative computing.

[0055] The second server 130 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In an embodiment of the present application, the second server 130 is used to support an application in a three-dimensional virtual environment to provide an action decision service for a virtual character. When the action decision service provided by the second server 130 is turned on, its own address is registered in ETCD (et-see-dee, a distributed key-value pair storage). In the case of an action decision requirement for a virtual character, the first server 120 requests the service dispatch to connect to the inference service, and obtains the address of the action decision service returned by the service dispatch. Thus, the first server 120 sends an action decision request to the action decision service provided by the second server 130 through the obtained address, and after the action decision service determines the target action based on the trained action decision model, it returns the action instruction to the first server 120.

[0056] During the process of training the action decision model, the second server 130 trains the action decision model through reinforcement learning based on the state information sent by the first server 120 .

[0057] The embodiments of the present application can be applied to scenarios where virtual characters navigate a virtual environment, scenarios where virtual actions complete a set task, or scenarios where a virtual character competes with other virtual characters, and the present embodiment is not limited thereto. The following is a schematic illustration of the application process of the embodiments of the present application in a virtual scenario.

[0058] The virtual character action decision method provided in the embodiments of the present application is applied in a game scenario. When an action decision is to be made for a first virtual character in the game scenario, the terminal 110 or the first server 120 obtains the state information of the first virtual character's game. The action decision model trained by the second server 130 obtains n target sub-actions and target actions serially output by n action output heads based on the state information. During the training process, sample state information is obtained and input into the action decision module to train the action decision model.

[0059] It should be noted that the embodiments of the present application can also be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, assisted driving, etc. The above implementation environment is only an illustrative example and does not limit the application scenarios of the embodiments of the present application.

[0060] For the convenience of description, the following embodiments are described by taking the action decision method of a virtual character in a virtual scene executed by a computer device as an example.

[0061] Please refer to Figure 2, which shows a flowchart of a method for action decision-making of a virtual character provided by an exemplary embodiment of the present application. This embodiment is described by taking the method executed by a computer device as an example, and the method includes the following steps:

[0062] Step 201: Obtain status information.

[0063] The status information is used to represent the game status of the first virtual character.

[0064] Optionally, the state information may include the first virtual character's state information during the game, the first virtual character's environmental perception information of the virtual environment, and other information that can represent the game scene state. The character state information is a one-dimensional vector, and the environmental perception information is a two-dimensional image.

[0065] Among them, character status information refers to information that can reflect the current status of the first virtual character, which may include the current attribute information of the first virtual character, interaction information generated by the interaction between the first virtual character and other virtual characters (used to characterize the interaction status between the first virtual character and other virtual characters in the game), and interaction information between the first virtual character and the environment (used to characterize the interaction status between the first virtual character and the virtual environment in the game).

[0066] In a gaming scenario, the computer device can obtain current status information during each game frame, or at regular intervals. The more game frames per second, the smoother the interface display. For example, if the game frame rate is 36, the computer device will obtain status information for 36 frames per second. For a higher frame rate, such as 72, where higher performance is required, the period can be set to 3, meaning one frame of status information is obtained every three game frames, for a total of 24 frames per second.

[0067] In a possible implementation, the character state information is a one-dimensional vector, that is, a vector feature. In the game scene, the character state information can be directly obtained from the game engine interface. In addition, part of the character state information can also be obtained through ray detection.

[0068] For illustration, please refer to Table 1, which shows the classification of role status information provided by an exemplary embodiment of the present application.

[0069] Table 1

[0070] In some embodiments, the character state information includes a variety of scalar information describing the current situation, including information about the state of the first virtual character and information about the first virtual character's interaction with the environment. Essentially, environmental perception information is derived from the first virtual character's perception of the surrounding virtual environment, characterizing the relative characteristics of the first virtual character relative to the virtual environment. Therefore, training the action decision model in a sufficiently rich virtual environment ensures that the relative characteristics captured by the action decision model are sufficiently rich and diverse, thereby ensuring the action decision model's generalized ability to perceive environmental perception.

[0071] However, the character status information may contain some specific features that are strongly associated with a specific map scene, so the computer device will perform further generalization processing based on the features.

[0072] Optionally, the computer device obtains relative state information by generalizing absolute state information included in the character state information, where the relative state information is state information of the first virtual character relative to the second virtual character or the obstacle.

[0073] Specifically, generalization processing can be achieved by replacing absolute information with relative information. For example, the absolute position information or absolute position orientation of the first virtual character, the second virtual character, and the cover point can be replaced with the relative position, that is, the position orientation, between the first virtual character and the second virtual character, and between the first virtual character and the cover point or obstacle.

[0074] The character state information includes interaction information between the first virtual character and the virtual environment, and the interaction information can be obtained by ray detection.

[0075] Optionally, environment perception rays are emitted in all directions, starting from the first virtual character's location. Upon impact with an obstacle surface, the environment perception rays are reflected along the normal direction of the environment perception rays. The computer device can then obtain the reflection information of the environment perception rays to determine character state information regarding the interaction between the first virtual character and the virtual environment. For example, the interaction information between the first virtual character and the cover and the interaction information between the first virtual character and the obstacle are shown in Table 1.

[0076] Schematically, Figure 3 shows a schematic diagram of a ray detection scheme provided by an exemplary embodiment of the present application. The computer device uses the waist of the first virtual character as the starting point and emits environmental perception rays in all directions. When the environmental perception rays collide with obstacles in the virtual environment, they are reflected, thereby obtaining interaction information between the first virtual character and the obstacles based on the reflection of the environmental perception rays.

[0077] Optionally, to more accurately obtain interaction information between the first virtual character and the obstacle, the computer device emits environment perception rays in all directions, starting from at least two heights of the first virtual character's location. The computer device generates interaction information between the first virtual character and the obstacle based on reflections of the environment perception rays.

[0078] The different environmental perception rays emitted at the same height are emitted in the same horizontal direction. For example, starting from the feet, waist, and head of the first virtual character, environmental perception rays (i.e., circular rays) are emitted in all directions to obtain interaction information between the first virtual character and obstacles.

[0079] In one possible implementation, the number of environmental perception rays emitted from different altitudes is the same, that is, the number of annular rays in each layer is the same. The reflection patterns of the annular rays at different altitudes can represent the distances between obstacles in the virtual environment and the first virtual character at different altitudes. The greater the number of annular rays in each layer, the more refined the perception of obstacles around the first virtual character, and the higher the performance requirements of the computer equipment.

[0080] Optionally, the environmental perception information in the game is a two-dimensional depth map of the direction in which the first virtual character is facing, and the two-dimensional depth map is used to represent the depth of obstacles in the direction in which the first virtual character is facing.

[0081] In a possible implementation, the computer device acquires environmental perception information through ray detection based on the position of the first virtual character in the virtual environment and the orientation of the first virtual character.

[0082] Specifically, an environment perception ray is emitted in the direction of the first virtual character, starting from the first virtual character. When the environment perception ray hits an obstacle surface, it is reflected along the normal direction of the environment perception ray, so that the computer device can generate environment perception information based on the reflection of the environment perception ray.

[0083] For illustration, please refer to FIG4 , which shows a schematic diagram of a two-dimensional depth map provided by an exemplary embodiment of the present application. The schematic diagram of the virtual environment corresponds to the two-dimensional depth map, wherein the number in each square represents the grayscale value of the area in the image, and the grayscale value ranges from 0 to 255, with white being 255 and black being 0. The smaller the grayscale value, the darker the pixel color in the image area. The environmental perception information is a two-dimensional depth map in the direction of the first virtual character. The darker the pixel color in the depth map, the closer the obstacle corresponding to the pixel is to the first virtual character. The lighter the pixel color in the depth map, the farther the obstacle corresponding to the pixel is from the first virtual character.

[0084] Optionally, the degree of refinement of the virtual environment represented by the two-dimensional depth map is positively correlated with the resolution. The greater the sampling resolution of the two-dimensional depth map, the finer the depth information of the obstacle in front of the first virtual character represented by the two-dimensional depth map.

[0085] Step 202: input the state information into the action decision model to obtain n target sub-actions serially output by n action output heads in the action decision model.

[0086] Among them, different action output heads correspond to different action types, and n action output heads are connected in series based on the dependency relationship between action types. The dependency relationship is used to characterize the dependency restrictions between sub-actions under different action types, and n is a positive integer.

[0087] In the action decision model, n action output heads belong to the output layer of the action decision model, which are used to predict actions in different dimensions, thereby outputting action parameters of different dimensions. Among them, the action dimensions supported by the first virtual character may include movement state dimension, movement direction dimension, orientation dimension, attack dimension, and posture dimension, etc. In the action decision model, n action output heads correspond to n action dimensions respectively, and then the n action output heads output action parameters of n different dimensions. For example, the action parameter output by the action output head corresponding to the movement direction dimension is a specific movement direction, such as a 90° direction; the action parameter output by the action output head corresponding to the orientation dimension is a turning angle that can represent the direction, such as -5°, representing a 5° rotation in the counterclockwise direction; the action parameter output by the action output head corresponding to the attack dimension is attack or not attack, etc. This embodiment does not limit the specific division of action dimensions and the action parameters output by the action output heads corresponding to different action dimensions.

[0088] Optionally, the computer device obtains n action types by orthogonally decomposing the executable actions of the first virtual character, where different action types contain at least two executable actions, and the sub-actions under different action types can be controlled independently, that is, the first virtual character can be controlled to only execute sub-actions in one action type.

[0089] Among them, orthogonal decomposition is used to divide the executable actions of the first virtual character into multiple discrete action types. The n action types obtained after orthogonal decomposition are mutually orthogonal action types. There is no intersection between the sub-actions in different orthogonal action types, that is, there is no situation where a sub-action belongs to two action types. Different orthogonal action types support simultaneous execution, and different sub-actions in the same orthogonal action type do not support simultaneous execution.

[0090] In the game scene, there are the most basic and fine-grained atomic actions to control the first virtual character, and such actions cannot be further split and refined. The sub-actions in an action type are atomic actions, and different sub-actions in the same action type cannot be executed at the same time. For example, if the first virtual character can perform actions including crawling forward, this action can be divided into two atomic actions, crawling and moving forward, where crawling corresponds to the posture action type, and moving forward corresponds to the moving direction action type. For another example, if the first virtual character can perform actions including standing still and peeking and shooting, this action can be divided into three atomic actions, staying in place, peeking, and shooting, where staying in place corresponds to the moving state action type, peeking corresponds to the peeking action type, and shooting corresponds to the attacking action type.

[0091] For illustration, please refer to Figure 5, which shows a user interface diagram of an exemplary application providing a virtual environment provided by this application. The user interface includes a movement state control 501, an attack control 502, a posture adjustment control 503, a horizontal and vertical steering control 504, a left and right probe control 505, and a movement state control 506. Different controls can trigger different actions of the virtual character.

[0092] By orthogonally decomposing the actions corresponding to the various controls shown in the above figure, multiple action types can be obtained.

[0093] In some embodiments, dependencies may exist between different action types. For example, a virtual character will only stand or sideways when performing an attack action, so the first virtual character's posture action decision needs to depend on the action decision of the attack action type. Furthermore, dependency restrictions indicate that there may be dependency restrictions between sub-actions within different action types. For example, when the first virtual character performs a sprint action, it does not support simultaneous execution of an attack action, so there is a dependency restriction between the sprint action and the attack action. Therefore, in order for the action decision model to consider the dependencies between different actions and the dependency restrictions between different sub-actions within different action types when making action decisions, the computer device serially connects the n action output heads in the action decision layer based on the dependencies between action types.

[0094] Optionally, when determining the order in which different action output heads are connected, the computer device may sort the order of the action output heads based on the degree of relevance of different action types to the task to be completed by the first virtual character. For example, if the first virtual character's task is to defeat an enemy, it may be determined that this task is highly associated with an attacking action type, and the corresponding action output head may be arranged at the front; whereas if this task is significantly less associated with a sideways action type, the action output head corresponding to the sideways action type may be arranged at the back.

[0095] Step 203: Control the first virtual character to perform a target action consisting of n target sub-actions.

[0096] In some embodiments, the action decision model is deployed in the second server. When the target action is determined by the action decision model, the second server sends the action execution instruction to the first server that provides background services for supporting the application of the virtual environment, and the first service server then sends the action execution instruction to the client to achieve control of the execution of the action by the first virtual character.

[0097] In one possible implementation, a computer device encodes the names of each sub-action in different action types separately to obtain multiple candidate action labels. The computer device obtains the probability distribution of each sub-action under each action type based on the action decision layer in the action decision model, and determines the corresponding label of the target sub-action from the sub-action label based on the action execution probability. When the target sub-action label is determined, the computer device decodes the target sub-action label to obtain the target sub-action, and controls the first virtual character to perform the target action composed of multiple target sub-actions.

[0098] In summary, in the embodiments of the present application, action decisions are made based on state information through an action decision model, thereby being able to determine a reasonable target action based on the current state. The action decision model includes n action output heads connected in series, each of which is used to output different target sub-actions. The n action output heads are connected in series based on the dependencies between action types. This allows the dependencies between different actions to be considered when making action decisions, and in the process of serially outputting the n target sub-actions, the n target sub-actions are causally related in time sequence, thereby improving the rationality of the target action and enhancing the anthropomorphism of the virtual character.

[0099] In the embodiment of the present application, the action decision model includes an information processing layer, a feature extraction layer, and an action decision layer, and the action decision layer includes n action output heads. The process of determining n target sub-actions will be described below through an exemplary embodiment.

[0100] Please refer to FIG6 , which shows a flowchart of a process for determining a target action provided by an exemplary embodiment of the present application.

[0101] Step 601: Obtain status information.

[0102] The implementation of this row of steps can refer to the above step 201, and this embodiment will not elaborate on it.

[0103] Step 602: Input the state information into the action decision model, and encode the state information through the information processing layer to obtain a state code.

[0104] Optionally, the state information includes both character state information of the first virtual character in the game and environmental perception information of the virtual environment in which the first virtual character resides. The character state information represents the interaction state between the first virtual character and the second virtual character in the game, as well as the interaction state between the first virtual character and the virtual environment. The character state information is a one-dimensional vector, and the environmental perception information is a two-dimensional image. In some embodiments, the computer device encodes the character state information and environmental perception information contained in the state information using different encoding methods through an information processing layer.

[0105] Optionally, the information processing layer includes a scalar encoder and an image encoder. The computer device inputs state information into the action decision model. The scalar encoder in the information processing layer encodes the character state information to obtain a state information encoding result. The image encoder in the information processing layer then encodes the environmental perception information to obtain an environmental information encoding result. Finally, the information processing layer concatenates the state information encoding result and the environmental information encoding result to obtain a state code.

[0106] Optionally, the scalar encoder includes a one-dimensional convolution kernel. The computer device performs one-dimensional convolution processing on the attribute information and interaction information in the character state information through the one-dimensional convolution kernel to obtain a state information encoding result.

[0107] Optionally, the image encoder includes a two-dimensional convolution kernel. The computer device encodes the environmental perception information in the state information using the two-dimensional convolution kernel to obtain an environmental information encoding result. The two-dimensional convolution kernel is a two-dimensional matrix. During the process of encoding the two-dimensional depth map using the two-dimensional convolution kernel, the feature encoding of the environmental perception information is completed by element-by-element multiplication of the two-dimensional matrix and the input two-dimensional depth map and summing the results.

[0108] After obtaining the state information encoding results and the environmental information encoding results, the computer device concatenates the state information encoding results and the perception information encoding results through a fully connected layer and an activation function to obtain the state encoding. Among them, the fully connected layer is used to map the feature space calculated by the previous convolution layer to the sample label space (that is, to integrate the feature representation into a single value), which can reduce the influence of feature position on the classification result and improve the robustness of the action decision model. The activation function is used to introduce nonlinear factors, thereby improving the expressive power of the action decision model.

[0109] In step 603, the state code is input into the feature extraction layer, and the state code features are extracted by the feature extraction layer to obtain fusion features.

[0110] In one possible implementation, the information processing layer also includes an LSTM (Long Short Term Memory) network. The computer device inputs the state code into the LSTM network and extracts the resulting fused features through feature extraction. During the feature extraction process, the LSTM network retains the more valuable features and discards the less valuable ones, thereby achieving feature extraction.

[0111] Optionally, the LSTM network can be replaced by other convolutional neural networks, which is not limited in this embodiment.

[0112] Step 604: Based on the fused features, n target sub-actions are determined through the n action output heads in the action decision layer.

[0113] In some embodiments, the computer device inputs the fused features into the action decision layer, and since the n action output heads in the action decision layer are connected serially, inputting the fused features into the action decision layer is to first input the fused features into the first action output head of the action decision layer, and determining the first target sub-action from the sub-actions in the first action type through the first action output head in the action decision layer.

[0114] Optionally, the action decision model further includes an information filtering layer, which is connected to the output of the scalar encoder and the input of the action decision layer. Upon acquiring the character state information, the computer device filters the character state information through the information filtering layer to obtain filtered character state information, wherein the filtered character state information is correlated with the first action type corresponding to the first action output head. Furthermore, the computer device inputs the filtered character state information into the first action output head. Furthermore, since the first action output head has the highest degree of correlation with the task to be completed, the character state information with the highest correlation with the first action type is input into the action decision layer together with the fusion feature, thereby enhancing the reasonable decision-making ability of the first action output head.

[0115] Illustratively, Table 2 shows the correspondence between action output heads and action types after orthogonal decomposition provided by an exemplary embodiment of the present application.

[0116] Table 2

[0117] In one possible embodiment, since different action output heads are connected in series, after the computer device inputs the fused features into the first action output head in the action decision layer and determines the first target sub-action from the first action type through the first action output head, it also needs to input the embedded coding vectors corresponding to the determined first target sub-action to the i-1th target sub-action, as well as the fused features into the action decision layer, and determine the i-th target sub-action from the i-th action type through the i-th action output head in the action decision layer.

[0118] The first target sub-action to the (i-1)th target sub-action are respectively determined by the first action output head to the (i-1)th action output head, where i is less than or equal to n and i is greater than 1.

[0119] Optionally, the n action output heads are connected via an autoregressive embedding layer. Because different action types have dependencies, after the preceding output head determines the target sub-action, the at least one determined target sub-action and the fused features should be input into the subsequent action output head based on the autoregressive embedding layer, so that the subsequent action output head can determine the target sub-action based on the dependencies.

[0120] Autoregression is a statistical method for processing time series, using the performance of the same variable at previous moments to predict the current moment. Autoregression exhibits autocorrelation in time series. Embedding is essentially data compression, using lower-dimensional features to represent higher-dimensional features with redundant information.

[0121] Please refer to Figure 7, which shows a schematic diagram of the autoregressive structure of n action output heads provided by an exemplary embodiment of the present application. An action output head includes a fully connected layer, a Logits layer, a sampling layer, and an embedding layer. The Logits layer is the execution probability distribution between each sub-action in the determined action type, thereby determining the target sub-action. After the target sub-action is determined, the determined sub-action is transmitted to the next serially connected action output head through the embedding layer, and the action output head fuses the target sub-actions that have been determined in the previous sequence, thereby determining a new target sub-action based on the determined target sub-action. In addition, the information processing layer also inputs the fused features into each action output head.

[0122] In this embodiment, the action control is discretized by orthogonal decomposition to reduce its dimensionality. The target sub-actions are then serially determined based on the n action output heads in the action decision model. This effectively decouples the structured action space. Furthermore, the combination of serial action output heads improves the rationality of the action decision model and increases training efficiency during model training.

[0123] In some embodiments, in the process of serially outputting n target sub-actions through n action output heads, taking into account the dependency and restriction relationships between the sub-actions under different action types, in order to shield unreasonable sub-actions in different action types, or sub-actions that have dependency and restriction relationships with the determined target sub-actions, the computer device can also set an action mask layer in the action decision model to mask the sub-actions in different action types through the action mask layer.

[0124] Essentially, action masking is used to screen out sub-actions of different action types that are unreasonable or have a dependent or restrictive relationship with the determined target sub-action. Action masking reduces the probability of the masked sub-action in the probability distribution obtained by the action output head, so that when the target sub-action is determined based on probability sampling, the masked sub-action is difficult to be determined as the target sub-action. By using action masking, unnecessary sub-action branches can be directly eliminated during the action decision process, thereby reducing the action space and improving the exploration efficiency of the action decision model while ensuring the output accuracy of the action decision model.

[0125] Among them, at least two sub-actions with a dependency restriction relationship belong to different action types, and at least two sub-actions with a dependency restriction relationship cannot be executed at the same time. Optionally, if there is a dependency restriction relationship between the first sub-action and the second sub-action, the two sub-actions cannot be executed at the same time. Therefore, when the first sub-action has been determined to be the target sub-action, it is necessary to perform action masking on the second sub-action. Optionally, if there is a dependency restriction relationship between the second sub-action and the third sub-action after the second sub-action is combined with the third sub-action, and the fourth sub-action, the combination of the second sub-action and the third sub-action and the fourth sub-action cannot be executed at the same time. Therefore, when it is determined that the second sub-action and the third sub-action are both target sub-actions, it is necessary to perform action masking on the fourth sub-action.

[0126] In the related art, when applying action masks to parallel action output heads, each action output head outputs sub-actions in parallel, and the action output heads are connected in parallel. Each action output head outputs different sub-action instructions. For the action output head that is executed with the action mask, the probability of its output action being selected is close to 0.

[0127] Please refer to Figure 8, which shows a schematic diagram of a related art method for performing action masking on a parallel action output head. During action masking, in the network's original logits action output layer, a large negative number is added to the logits of unreasonable actions to ensure that the logits value corresponding to the action is smaller than the logits value of all reasonable actions, thereby reshaping the logits distribution of the output action layer and mapping the probability of unreasonable actions being sampled close to zero. In the figure, a1 to an+1 represent the logits distribution of different actions, and P1 to Pn+1 represent the sampling probability corresponding to each action. In the case of i=3 in the corresponding figure, the action corresponding to a3 is masked and a large negative number is added to obtain a new logits distribution. When sampling again according to the probability, it is difficult to sample the action corresponding to a3.

[0128] Optionally, since sub-actions under different action types have dependency restriction relationships, and at least two sub-actions with dependency restriction relationships do not support simultaneous execution, then in the case of a determined preceding target sub-action, some sub-actions in the subsequent action type may have dependency restriction relationships with the determined target sub-action. Moreover, since the dimension of the action space after orthogonal decomposition is still very high, it will lead to a rich explosion of anthropomorphic atomic action space combinations. Therefore, sub-actions can be determined based on the rationality of the combination between different sub-actions (i.e., dependency restriction relationships), thereby improving the anthropomorphism of the action combination. In addition, in the process of training the action decision model, action masking can also be used to reduce invalid decision strategy exploration at the action decision layer, thereby accelerating training efficiency and improving the anthropomorphism of the action decision model's action decisions.

[0129] Action masks can be used in reinforcement learning models (corresponding to the action decision model in the embodiment of the present application). They can shield unreasonable or invalid action sets in large-scale decision spaces, thereby reducing meaningless action exploration and enabling the action decision layer to converge faster.

[0130] Optionally, the action decision model includes an action mask layer, which is connected to n action output heads in the action decision layer and is used to perform action masking on sub-actions in each action type in the action decision process of each output head.

[0131] In one possible implementation, the computer device performs action masking on the sub-actions within the i-th action type based on the determined first through i-1th target sub-actions, as well as the dependency constraints indicated by the dependency constraints between sub-actions within different action types. At least two sub-actions with the dependency constraints cannot be executed simultaneously. The computer device then determines the i-th target sub-action from the unmasked sub-actions within the i-th action type using the i-th action output header in the action decision layer.

[0132] Optionally, before action masking, the computer device first determines, based on the determined target sub-action, among the action types to be decided, sub-actions to be masked that have a dependency constraint relationship with the determined target sub-action. Specifically, the following three situations may be involved:

[0133] 1. There is a determined target sub-action that has a dependency restriction relationship with the sub-action to be masked in the action type to be decided.

[0134] In one possible embodiment, when there is a determined j-th target sub-action and the dependency restriction situation indicates that the j-th target sub-action has a dependency restriction relationship with at least one first sub-action in the i-th action type, the computer device performs action masking on the first sub-action in the i-th action type, and j is less than i (i.e., the j-th action head is connected in series before the i-th action head).

[0135] For example, if there is a determined sprinting action, and the dependency restriction situation indicates that there is a dependency restriction relationship between the sprinting action and the squatting action, then when determining the action type of the first virtual character posture, the computer device can perform action masking on the squatting action in the posture action type.

[0136] Optionally, the computer device can perform one-hot processing on the target sub-action determined by the output head of the preceding action, and then obtain the action mask matrix corresponding to the subsequent action type by matrix multiplication with the one-hot matrix corresponding to the preceding target sub-action based on the restriction matrix used to characterize the dependency restriction relationship between each sub-action in the subsequent action type and the sub-action in the preceding action type.

[0137] The specific method is as follows: the computer device transmits the action determined by the previous action output head to the subsequent action output head based on the action mask matrix.

[0138] Schematically, there are action type a1 for initiating an attack (including attacking and not attacking), action type a2 for the avatar's posture (including squatting, jumping, crawling, and standing), and action type a7 for the avatar's movement (including standing still, walking fast, sprinting, and standing still). Action type a1 corresponds to action output head H1, action type a2 corresponds to action output head H2, and action type a7 corresponds to action output head H7. Attacking and sprinting have a dependency constraint, and squatting also has a dependency constraint on sprinting. Furthermore, because the values ​​of sub-actions are transmitted in the neural network as tensors, it is not possible to directly sample the target sub-action to avoid gradient disconnection.

[0139] Optionally, the action mask matrix corresponding to each action output head can be determined during the training process of the action decision model. Taking the action output heads H1, H2 and H7 as an example, when there are three output sample target actions, the computer device first constructs a one-hot matrix τ based on the three sample target actions. n×m , where n = 3, indicating that there are three sample target actions, and m = 2, indicating the action dimension of a1 (i.e., the number of sub-actions included).

[0140] Optionally, when the three sample target actions obtained are output from the action output head H1 as {attack, do not attack, attack}, the corresponding n×m one-hot matrix τ n×m It can be expressed as:

[0141] τ n×m It is the one-hot matrix output by the action output head H1 in the sample target action. The number of rows is the number n of sample target actions, and the number of columns represents the number m of sub-actions contained in the action type a2.

[0142] The computer then constructs an auxiliary mapping matrix M, specifically a mapping matrix that characterizes the dependency constraints between the sub-actions of action type a1 and the sub-actions of action type a7. Since the mapping relationship between action type a7 and the target sub-action samples output by a single action output head H1 is constructed based solely on the target sub-action samples, the one-hot matrix composed of all the different actions of action output head H1 can be determined as an m×m identity matrix, with the first row mapping the target sub-actions to attack, and the second row mapping the target sub-actions to not attack.

[0143] Since the action output head H7 corresponds to dimension P = 4, in order to reflect the attack without sprinting, the auxiliary mapping matrix M can be obtainedm×p for:

[0144] M m×p The essence of the matrix is ​​a mapping from H1 to H7. The number of rows represents the dimension m of the preceding action head H1, and the number of columns represents the dimension P of the subsequent affected action head H7. Each row is related to the unit matrix E. m×m Each row in corresponds to an association, and the specific value of each row refers to the impact of each sub-action in the previous action output head H1 on the subsequent output action head H7.

[0145] Finally, the computer device multiplies τ by matrix multiplication. n×m The effect of not being able to sprint when launching an attack is mapped to the action output head H7, and the action mask matrix X7 corresponding to the action output head H7 is obtained as follows:

[0146] The action mask matrix X7 is used to characterize the influence of the sample target sub-action output by the preceding action output head H1 in all sample target actions on the sub-action in the action type corresponding to the subsequent action output head H7. It can be seen that the size of X7 is only related to the number of samples and the dimension of the action output head H7. The number of rows represents the sampling of 3 samples, and the number of columns represents the different sub-actions in the action type a7. The above process has masked the sprint sub-action corresponding to the attack when the target sub-action determined in all sample target actions is attacked, and then the probability distribution of the sub-action in the action type a7 determined by the action output head H7 is converted into a probability distribution, which can ensure that the action output head H7 will not sample the sprint action.

[0147] Similarly, the computer device also uses the above processing method to obtain the action mask matrix Y7 of the target sub-action in action type a2 on action type a7. Since action type a1 and action type a2 both independently affect action type a7, the action output head H7 superimposes the influence of the target sub-action output by the previous action on the sub-action in action type a7, thereby obtaining the action mask matrix L7 corresponding to the action output head H7:

[0148] The L7 matrix is ​​the Hadamard product of X7 and Y7. All sub-actions in action type a7 mapped by all elements with a value of 0 in L7 will be masked.

[0149] Based on the above method, by mapping the influence of the target sub-action determined in the previous order on the sub-action in the action type corresponding to the subsequent action output head to the action mask matrix, and then transferring the influence backward in the n action output heads in sequence, it is possible to effectively solve the problem of a large amount of invalid exploration of the decision model in the high-dimensional action space during the training process of the action decision model.

[0150] 2. There are at least two determined target sub-actions that, after being combined, have a dependency restriction relationship with the sub-action to be masked in the action type to be decided.

[0151] In another possible implementation, there may be determined target sub-actions, and these two target sub-actions do not have an independent impact on the sub-actions in the subsequent action type. After at least two determined target sub-actions are combined, there will be a dependent restriction relationship with the sub-actions in the action type to be decided.

[0152] Schematically, please refer to Figure 9, which shows a schematic diagram of the influence of the moving direction and the heading direction superimposed on the moving state provided by an exemplary embodiment of the present application. There are two action types: moving direction and horizontal steering angle. The sub-actions in these two action types will not affect the sprint action in the determined action type alone. However, when the difference between the horizontal steering angle and the moving direction in the determined target sub-action is large (180° in the extreme case), as shown in the figure, the gaze direction of the first virtual character differs by 180° from the moving direction. Since the first virtual character is not supported to sprint backward in the conventional setting, the first virtual character does not support the simultaneous execution of the sprint action.

[0153] In this case, the computer device can combine the target sub-actions output by at least two action output heads to determine the impact of the combination of at least two target sub-actions on the sub-actions in the action type to be decided, and map the impact into an action mask matrix corresponding to the action type to be decided.

[0154] Specifically, the computer device may first encode the target sub-actions output by the at least two action output heads into one-hot matrices, then concatenate the at least two one-hot matrices to form a merged one-hot matrix. The merged one-hot matrix is ​​then logically transformed to map it onto a unit matrix. Subsequently, based on the method described in Method 1 above, an auxiliary mapping matrix is ​​determined. Based on the auxiliary mapping matrix, the impact of the combination of the at least two target sub-actions on the action type to be decided is determined, thereby implementing action masking of the sub-actions within the action type to be decided.

[0155] Schematically, there is action type a3 for movement direction (including eight movement directions evenly distributed within 360°), action type a4 for horizontal orientation (including eight orientation angles evenly distributed within 360°), and action type a7 for the movement state of the virtual character (including still walking, fast walking, sprinting, and standing still). Action type a3 corresponds to action output head H3, action type a4 corresponds to action output head H4, and action type a7 corresponds to action output head H7.

[0156] Among them, the action type a3 corresponds to the action dimension s, the action type a4 corresponds to the action dimension h, and all different sampling sub-actions of the action output head H3 are combined into a unique hot matrix s×s unit matrix E s×s , the one-hot matrix composed of all different sampled actions of the action output head H4 is an h×h identity matrix E h×h The computer device is calculated by the identity matrix E s×s and the identity matrix E h×h Feature concatenation, the merged single-hot matrix T can be obtained (h×h)×(s+h) It is not a square matrix. The number of rows is s×h, which represents all action combinations. The number of columns is s+h, which represents the combined dimension, i.e., the sum of the number of sub-actions of action type a3 and action type a4. (s×h)×(s+h) Converted to an identity matrix E of dimension s×h (s×h)×(s×h) , then each row of the identity matrix represents a combination of actions, and then the mapping auxiliary matrix M can be determined according to the dependency restriction relationship.

[0157] Optionally, assume that the merged unique-hot matrix corresponding to the sample target sub-actions output by the action output head H3 and the action output head H4 in all sample target actions is τ n×(s+h) , the action mask matrix corresponding to the action output head H7 can be determined by the following calculation:

[0158] First, calculate the mapping of each action combination in the sample target action into an approximate one-hot matrix: U n×(s×h) =τ n×(s+h) ×X (s+h)×(s×h)

[0159] Among them, X (s+h)×(s×h) T (s×h)×(s+h) The generalized inverse matrix, U n×(h×h) The mapping of each action combination in the sample target action into an approximate one-hot matrix, since X (s+h)×(s×h) Not T (s×h)×(s+h) The exact inverse matrix requires U n×(s×h) Convert to an exact one-hot matrix: Z n×(s×h) =one-hot(argmax(U n×(s×h) ))

[0160] where Z n×(s×h) Each row represents an action combination of a sample. Each row contains s×h matrix values, with only one element value being 1 and the rest being 0. The element value of 1 indicates that the action combination will be associated with the corresponding row of the mapping auxiliary matrix M, thereby affecting the matrix value of the corresponding row of the action mask matrix corresponding to the action output head H7. n×pis the action mask matrix in the action output head H7, and each row represents the impact of action type a3 and action type a4 on the sub-actions in action type a7.

[0161] Through the above calculations, action masking can be achieved when the action combinations between different sub-actions have a dependent and restricted relationship with the sub-actions in the decision action type, thereby solving the problem of low exploration efficiency of the action decision layer in a strongly coupled high-dimensional complex action space.

[0162] 3. The character status information and environmental perception information indicate the current virtual environment situation, and have a dependency restriction relationship with the to-be-masked sub-action in the to-be-decided action type, that is, it is not suitable to execute the to-be-masked sub-action in the current virtual environment.

[0163] In one possible implementation, the first virtual character may have a task objective such as attacking the second virtual character or avoiding the second virtual character. Therefore, when making an action decision, the computer device may treat the sub-action action mask in the decision action type based on the visibility of the first virtual character to the second virtual character.

[0164] Specifically, the computer device emits an environmental perception ray, starting from the eye position of the first virtual character, toward various parts of the second virtual character in the direction of the first virtual character. Based on the reflection of the environmental perception ray, the visibility of various parts of the second virtual character in the virtual environment is determined. The visibility is used to indicate whether various parts of the second virtual character are obscured by cover. If the environmental perception ray is obscured by an obstacle, the part of the second virtual character is not visible. If the environmental perception ray is not obscured by an obstacle and collides with the second virtual character, the part of the second virtual character is visible.

[0165] Please refer to Figure 10, which shows a schematic diagram of visual situation perception provided by an exemplary embodiment of the present application. A computer device emits environmental perception rays starting from the eyes of a first virtual character 1001 to various parts of a second virtual character 1002. If some of these perception rays are blocked by an obstacle 1003, the body parts of the second virtual character 1002 corresponding to these rays will not be visible. However, if some perception rays 1004 are not blocked by the obstacle and collide with the head of the second virtual character 1002, the head of the second virtual character 1002 will become visible.

[0166] After determining the visibility of each part of the second virtual character, the computer device masks the sub-actions in the target action type based on the visibility of each part of the second virtual character.

[0167] The target sub-action is dependent on the visibility of various parts of the second avatar. For example, if the target action is a shooting action, and the visibility of various parts of the second avatar indicates that the second avatar is not visible, the "shooting" sub-action in the target action is masked to prevent the first avatar from frequently shooting without aiming at the enemy.

[0168] After masking the sub-actions in the target action type, the computer device determines the sub-actions from the unmasked sub-actions in the target action type through the target action output head.

[0169] In another possible implementation, when the first virtual character needs to aim at the second virtual character, the computer device may first determine the range of steering angles within which the second virtual character can be aimed, and then perform action masking on the steering angles in the steering action type that prevent the first virtual character from aiming at the second virtual character.

[0170] Please refer to Figure 11, which shows a schematic diagram of determining the effective vertical aiming range provided by an exemplary embodiment of the present application. First, the distance d between the first virtual character and the second virtual character is determined based on the position coordinates of the second virtual character and the position coordinates of the first virtual character, and the height h of the second virtual character is obtained, where h = h_z - h_f. If the crosshair of the virtual prop held by the first virtual character falls within the height range of the first virtual character in the vertical direction, the second virtual character can be aimed at. According to trigonometric functions, we can obtain: θ max =ac tan[(h_z-h_f) / d] θ min =ac tan[(h_f-h_z) / d] τ pitch =θ max -θ min

[0171] Among them, the horizontal direction is the current direction, which is 0°, θ max is the effective aiming range of upward rotation, θ min is the effective aiming range of downward rotation, the difference between the two is τ pitch This is the effective aiming range for the numerical steering action.

[0172] For the horizontal turning angle, the computer device can determine the effective aiming range based on the distance d between the first virtual character and the second virtual character, and the width of the virtual characters, thereby masking the sub-actions in the horizontal turning action type, so that the first virtual character can aim at the second virtual character after performing the target sub-action in the unmasked sub-action.

[0173] Schematically, please refer to Figure 12, which shows a schematic diagram of the horizontal effective aiming range provided by an exemplary embodiment of the present application. The current orientation of the first virtual character is 0°, the distance between the first virtual character and the second virtual character is d, and the width of the second virtual character is W. The effective aiming range is determined as: θ = ac tan[(W / 2) / d] τ pitch =2θ

[0174] Where θ is an effective rotation range of the first virtual character in the clockwise and counterclockwise rotation angles. Combining the clockwise rotation effective range with the counterclockwise rotation effective range, the effective aiming range τ is obtained. pitch .

[0175] Subsequently, the computer device determines an effective horizontal turning angle based on the effective aiming range, and the determined effective horizontal turning angle is the orientation of the first virtual character after performing a turning action according to the fine-grained turning angle under the current orientation within the effective aiming range.

[0176] Furthermore, the computer device masks the steering angle actions that do not belong to the valid horizontal steering angles in the horizontal steering action, and determines the target horizontal steering angle from the valid horizontal steering angles that are not masked in the horizontal steering action through the horizontal steering action output head, thereby finally making the first virtual character perform the target horizontal steering action to aim at the second virtual character.

[0177] In one possible implementation, to prevent the target action from frequently performing a certain action, which is inconsistent with human behavior logic, the computer device can set a cooldown time for some sub-actions. If the cooldown of a sub-action has not yet completed, the sub-action will be masked. This prevents the first virtual character from repeatedly performing the same action and causing twitching.

[0178] In an embodiment of the present application, based on three different dependency restriction situations and the determined target sub-actions, the sub-actions in the action type to be decided are masked, thereby improving the action decision efficiency in high-dimensional complex action spaces and saving computing resources to a certain extent.

[0179] Please refer to Figure 13, which shows a schematic diagram of an action decision model provided by an exemplary embodiment of the present application. It includes an action mask layer, an information processing layer, a feature extraction layer and an action decision layer. The computer device encodes the acquired state information (character state information and environmental perception information) of the first virtual character through a scalar encoder and an image encoder, and then obtains the input feature extraction layer (LSTM network) after feature splicing through a fully connected layer and an embedding layer. The spliced ​​state features are subjected to feature extraction by the LSTM network to obtain fused features. After the fused features are input into the action decision layer, the target action is determined by the action decision layer. The target action consists of multiple target sub-actions, and the n target sub-actions are serially outputted by n action output heads respectively. In the process of determining the target sub-action, the action mask layer performs action masking on the sub-actions with dependency restriction relationships.

[0180] Furthermore, during model training, the action decision model also includes a value network, which evaluates the value of a specific target sub-action and adjusts the action decision model based on the resulting value estimate. Furthermore, partial game information can be fed into the value network to enable it to determine the value of a sample target sub-action based on the current game situation.

[0181] Before applying the action decision model, the action decision model should be trained in a training game. The following will illustrate the training process of the action decision model through an exemplary embodiment.

[0182] Please refer to FIG14 , which shows a flowchart of an action decision model training process provided by an exemplary embodiment of the present application. The process includes the following steps:

[0183] Step 1401: Obtain sample state information, where the sample state information is used to represent the game state of the training game in which the first virtual character is playing.

[0184] This step can refer to the process of obtaining status information in the above step 201, and will not be described in detail here.

[0185] Step 1402: Based on the sample state information, the action decision model is trained by reinforcement learning.

[0186] The process of training the action decision model based on reinforcement learning includes the following.

[0187] Step 1402a: Input the sample state information into the action decision model to obtain a sample target action output by the action decision model. The sample target action is composed of n sample target sub-actions.

[0188] In fact, the process of training the action decision model is the process of strengthening the training of the action decision layer in the action model.

[0189] The specific implementation of this process can refer to the above embodiment for determining the content of the target action, and this embodiment will not be described in detail here.

[0190] Step 1402b: Control the first virtual character to execute the sample target action, and obtain the sample action execution result of the first virtual character.

[0191] The sample action execution result includes the change in the state information of the game after the first virtual character executes the sample target action.

[0192] Step 1402c: Determine an action execution reward based on the sample action execution result of the first virtual character.

[0193] During the training process, corresponding action execution tasks are usually set for the first virtual character. Therefore, in order to avoid the first virtual character taking extreme operations to complete the target task, resulting in a low degree of anthropomorphism of the first virtual character, in addition to determining the execution reward of the target action based on the task progress, the computer device can also determine the execution reward of the sample target action based on the degree of anthropomorphism of the sample target action.

[0194] First, the computer determines the weight coefficient ratio for executing various reward and punishment items. Action execution rewards can be divided into primary rewards (i.e., rewards for progressing toward the target task) and auxiliary anthropomorphic rewards. During the initial training of the action decision model, the computer increases the weight of the primary reward and decreases the weight of the auxiliary anthropomorphic rewards, so that the action decision model prioritizes learning action decisions with the goal of completing the task. After multiple rounds of training, the strength of the action decision model continues to increase, and the computer needs to adjust the coefficient ratio of the auxiliary anthropomorphic rewards based on the performance of the action decision model and the trend of changes in the corresponding anthropomorphic index. The anthropomorphic index refers to the degree of match between the probability of the first virtual character performing a specific action and the probability of the real player performing the characteristic action. For example, if the first virtual character is unable to properly use cover to engage in a pull-and-pull battle during a game, it can be considered that the weight of the primary reward is too high, and the computer needs to reduce the weight of the primary reward to ensure that the action decision model can determine the anthropomorphic target action based on a certain strength.

[0195] Optionally, the information processing layer also includes a game information encoder, and the sample state information also includes sample game information, and the sample game information is used to characterize the real-time situation of the training game in which the first virtual character is located. In one possible embodiment, when the sample game state information is input into the action decision model, the computer device first encodes the sample game information through the game information encoder to obtain the game state code, and then determines the action execution reward based on the game state code and the sample action execution result. In this way, the action decision model can objectively judge the reward brought by the current target action based on the global game information, thereby reducing the variance of the value estimate and improving the training efficiency.

[0196] In an embodiment of the present application, the action execution reward mainly includes at least one of an attribute reward, a game victory reward, a game failure reward, and a task reward.

[0197] The attribute reward mainly includes the life attribute reward of the first virtual character. When the attribute value of the first virtual character decreases, the computer device can determine the attribute reward as an action execution reward. For example, after the first virtual character performs the sample target action, the life attribute of the first virtual character decreases, and it is determined that the target action obtains a negative attribute reward. Alternatively, when the attribute value of the first virtual character increases, the computer device can determine the attribute reward as an action execution reward. For example, after the first virtual character performs the sample target action, the life attribute of the first virtual character increases, and it is determined that the target action obtains a positive attribute reward.

[0198] The game failure reward means that when the first virtual character fails in the game, he will obtain a huge reverse reward (i.e., punishment). In the case that the first virtual character fails in the game, the computer device can determine the game failure reward as an action execution reward.

[0199] The reward for a successful game is the opposite of the reward for a failed game. When the first virtual character succeeds in the game, it will receive a huge positive reward. When the attribute value of the first virtual character decreases, the computer device can determine the attribute reward as an action execution reward.

[0200] In some embodiments, the first virtual character has a target task in the game, such as guarding a treasure chest, avoiding pursuit, or reaching a designated location, etc. Therefore, if the first virtual character succeeds in the task in the training game, the computer device can determine the task reward as an action execution reward.

[0201] In addition, in order to enable the action decision model to achieve anthropomorphic action decisions, the action execution reward also includes anthropomorphic rewards.

[0202] Optionally, the computer device determines anthropomorphic attribute values ​​for at least two sample target actions based on the sample action execution results of the first virtual character performing at least two sample target actions in the training game. The anthropomorphic attribute value may include the number of times a specific action is performed, the movement positions of at least two consecutively determined sample target actions, and the like. If the anthropomorphic attribute value is below an anthropomorphic attribute threshold, the computer device may determine an anthropomorphic reward as the action execution reward.

[0203] For illustration, please refer to Table 3, which shows the action execution reward provided by an exemplary embodiment of the present application.

[0204] Table 3

[0205] In the table above, some action rewards need to be determined based on multiple consecutive sample target actions. For example, a reasonable sideways movement is an anthropomorphic reward. Virtual characters controlled by real players usually do not sideways multiple times in a short period of time, or sideways when there is no cover around. Therefore, among the multiple sample target actions determined, if the anthropomorphic attribute value of the sideways movement (i.e., the number of sideways movements) exceeds the anthropomorphic attribute threshold, the computer device will determine it as an unreasonable sideways movement and reverse the reward based on the action decision model.

[0206] Table 3 includes rewards for dispersed movement, rewards for using cover, rewards for ambushing in the dark, and rewards for reasonable actions (reasonable sideways movement, reasonable squatting, reasonable still walking, etc.).

[0207] 1. Movement Dispersion Rewards: Rewards for reducing the closeness of movement for multiple consecutive times while the first avatar performs a sample target action. This means the first avatar will not wander around in the same place repeatedly, but will be able to move more dispersedly across large maps, exploring different locations.

[0208] Optionally, the sample action execution result includes the position point of the first virtual character after performing the first n+1 actions. Based on the position point of the first virtual character after performing the n+1th sample target action, the computer device determines a first closeness between the position point after performing the n+1th action and the position point after performing the previous n actions, and determines a second closeness between the position point after performing the nth action and the position point after performing the previous n-1 actions. If the first closeness is less than the second closeness, the computer device determines the movement dispersion reward as the action execution reward.

[0209] When training the action decision model, the first virtual character may be trapped in a circle in a local area. Therefore, in order to avoid this phenomenon, the computer device can guide the first virtual character to execute the sample target action determined by the action decision model based on the tight centrality algorithm, so that the movement can be sufficiently dispersed and the movement route can cover the entire map as much as possible, so as to train the movement ability of the first virtual character.

[0210] Specifically, if the degree of closeness between the position point after the n+1th action and the position point after the previous n actions is less than the degree of closeness between the position point after the nth action and the position point after the previous n-1 actions, the computer device determines that a positive movement dispersion reward has been obtained. Alternatively, if the degree of closeness between the position point after the n+1th action and the position point after the previous n actions is greater than the degree of closeness between the position point after the nth action and the position point after the previous n-1 actions, the computer device determines that a negative movement dispersion reward has been obtained. This guides the first virtual character to move as far away from the most recently visited historical locations as possible, avoiding the situation where the virtual character wanders in one place.

[0211] Second, the cover utilization bonus is the reward for the first avatar being adjacent to cover when performing the sample target action. In the presence of a second avatar, the first avatar is expected to stay close to cover as much as possible to avoid being attacked by the second avatar.

[0212] Optionally, the sample action execution result includes sample state information after the first virtual character performs the sample target action, including sample character state information and sample multi-environment perception information. The sample environment perception information includes a two-dimensional depth map, which is used to represent the obstruction of the first virtual character by obstacles in the direction of the first virtual character.

[0213] The computer device divides the two-dimensional depth map into at least two depth regions, and determines a first occlusion ratio for the first depth region and a second occlusion ratio for at least two second depth regions adjacent to the first depth region, where the first depth region is located at the center of the first virtual character's field of view. Furthermore, the computer device determines a cover utilization reward based on a minimum difference between the first occlusion ratio and the second occlusion ratio, and determines the cover utilization reward as an action execution reward, wherein the degree of the cover utilization reward is positively correlated with the minimum difference.

[0214] In the embodiments of the present application, the determination of cover utilization rewards is achieved by reshaping and modifying the depth map matrix. In a game scene, cover is an area that provides cover for the first virtual character during combat. Multiple cover exists in the virtual environment, and different cover differs in at least one of position, orientation, and shape. A two-dimensional depth map can represent the depth of obstacles facing the first virtual character. Therefore, a computer device can determine the pixel distribution area of ​​the cover based on the pixel value distribution of the two-dimensional depth map.

[0215] Optionally, the computer device divides the depth map numerically, obtaining 19 regions from 1 to 19. The depth map is then divided vertically into several columns, and the brightness ratio of each column is calculated. A higher brightness ratio indicates a brighter front, a lower likelihood of an obstruction, and a lower obscuration ratio. In applications, it is often desirable for the first virtual character to select an area with cover and be able to attack the second virtual character. Therefore, it is necessary to minimize the brightness ratio of the middle column of the depth map. The middle column reflects the visual field directly in front of the first virtual character. The lower the brightness ratio, the greater the obscuration ratio, and the higher the likelihood of a cover area directly in front. Furthermore, it is desirable for the first virtual character to be at the edge of cover, i.e., to be able to attack the second virtual character by moving sideways, etc. Therefore, it is necessary to ensure that the absolute value of the difference between the brightness ratio of the middle column and the larger brightness ratio of the two columns to its left and right is as large as possible, that is, to ensure that the minimum value of the difference between the brightness ratio of the middle column and the brightness ratio (obscuration ratio) of the two columns to its left and right is as large as possible. For example, it is desirable that the obscuration ratio of the 10th column is as large as possible, and that the obscuration ratio of an area between the 9th and 11th columns is as small as possible.

[0216] Optional, the brightness ratio v corresponding to the depth area j It can be calculated by the following formula:

[0217] where v j The ratio of the actual pixel in each depth region to the theoretical maximum pixel sum in each column, and the masking rate of the depth region is 1-v j The bunker utilization bonus can be calculated as follows: r = α·(1-v 10 )+β·|1-v 10 -min(1-v9,1-v 11 )|

[0218] Here, r is the cover exploitation reward, and α and β are the reward hyperparameters.

[0219] It should be noted that the two-dimensional depth map region division method can also be based on the horizontal division, or the number of divided regions can be adjusted based on the specific virtual environment. In the case of a more complex virtual environment, more depth regions can be used to perform a finer division of the two-dimensional depth map, thereby obtaining a finer cover effect. In the case of a large map scene and a relatively simple control structure, a coarse-grained depth region division can be performed, which is not limited in this embodiment.

[0220] 3. Ambush Bonuses are visibility bonuses. A visibility bonus is a reward given when the first virtual character, while performing a sample target action, is in a position that effectively attacks a second virtual character. If a second virtual character is present, it's desirable for the second virtual character to be visible to the first. However, if the first virtual character isn't visible to the second, it facilitates an attack.

[0221] Optionally, the sample action execution result includes a mutual visibility relationship between the first virtual character and the second virtual character, where the mutual visibility relationship is used to represent the visibility of the first virtual character and the second virtual character relative to each other.

[0222] Optionally, when the mutual visibility relationship indicates that the first virtual character is not within the visual range of the second virtual character, and the second virtual character is within the visual range of the first virtual character, the computer device determines that the action execution reward is a positive visibility reward.

[0223] In a case where the mutual visibility relationship indicates that the first virtual character is within the visible range of the second virtual character, and the second virtual character is not within the visible range of the first virtual character, the computer device determines that the action execution reward is a reverse visibility reward.

[0224] That is, when the first virtual character is in a favorable attacking position compared to the second virtual character, a positive visibility reward is determined to be obtained. Conversely, when the first virtual character is in a disadvantageous attacking position compared to the second virtual character, a negative visibility reward is determined to be obtained.

[0225] Optionally, the mutual visibility relationship can be obtained by ray detection. The computer device uses the eye position of the first virtual character as a starting point to emit environmental perception rays to various parts of the second virtual character in the direction the first virtual character is facing, and determines the visibility of various parts of the second virtual character in the virtual environment based on the reflection of the environmental perception rays. The visibility is used to characterize whether various parts of the second virtual character are blocked by cover. If the visibility indicates that various parts of the second virtual character are blocked by obstacles, it is determined that the second virtual character is not within the visible range of the first virtual character. Similarly, the computer device uses the eye position of the second virtual character as a starting point to emit environmental perception rays to various parts of the first virtual character in the direction the second virtual character is facing, and determines the visibility of various parts of the first virtual character based on the reflection of the environmental perception rays. If the visibility indicates that various parts of the first virtual character are blocked by obstacles, it is determined that the first virtual character is not within the visible range of the second virtual character.

[0226] 4. Reasonable Action Rewards refer to rewards given to the first avatar when it performs a sample target action that aligns with the real player's operating logic. During model training, we expect the first avatar's behavior to be as human-like as possible, performing actions that are more consistent with human logic.

[0227] Before determining the sample target action, the computer device determines an ideal action for the first virtual character based on the game information from the training game (including the sample character state information and the sample environmental perception information). For example, if the environmental perception information indicates that there is low cover in front of the first virtual character, the ideal action for the first virtual character may be a squat. For another example, if the environmental perception information indicates that there is a second virtual character to the right of the first virtual character, the ideal action may be a right turn.

[0228] Subsequently, when the sample target action corresponding to the action decision model is consistent with the ideal action, the computer device determines the reasonable action reward as the action execution reward.

[0229] Optionally, the computer device emits environmental perception rays in all directions with the first virtual character as the starting point, with different environmental perception rays located at the same horizontal height. The computer device determines the distribution of surrounding obstacles based on the reflection of the environmental perception rays. If there is a shortest environmental perception ray whose length is less than a distance threshold, it indicates that the first virtual character is close to the obstacle and it can be determined that a turn should be made at this time. The environmental perception rays diffuse in both directions with the shortest ray as the reference, and the computer device determines that the direction of the environmental perception ray whose length exceeds the distance threshold earlier is the ideal turning direction for the first virtual character.

[0230] When the turning direction determined by the action decision model is consistent with the ideal turning direction, the computer device determines the reasonable action reward as the action execution reward; when the turning direction determined by the action decision model is inconsistent with the ideal turning direction, the first virtual character may get stuck in the wall, and the computer device will give a certain reverse reward.

[0231] Step 1402d: Update the model parameters of the action decision model based on the action execution reward.

[0232] The process of updating the action decision model based on the action execution reward is to update the model parameters of the action decision model to obtain a larger action execution reward, so that the action decision model can achieve action anthropomorphism as much as possible under the premise of performing the target task.

[0233] In the embodiment of the present application, during the training of the action decision model, by adding auxiliary anthropomorphic action rewards, the dual requirements of model strength and anthropomorphism can be met, and combined with the action masking logic, the anthropomorphism of the virtual character is greatly improved, which is more in line with the behavioral logic of real players.

[0234] In an embodiment of the present application, a training game in which a first virtual character is present includes at least one second virtual character, and the second virtual character executes an action output by a second action decision model. In some scenarios, the objective of the second virtual character corresponds to the objective of the first virtual character. For example, the objective of the first virtual character may be to protect a treasure chest, while the objective of the second virtual character may be to capture the treasure chest.

[0235] The second virtual character serves as an opponent virtual character. After the first action decision model corresponding to the first virtual character is trained, the second action decision model also needs to be iteratively trained to obtain second action decision models of different strengths.

[0236] Specifically, when the first action decision model corresponding to the first virtual character completes the i-th round of training, the i-th round of training is performed on the second action decision model based on the trained first action decision model, and the second action decision model that has completed the i-th round of training is stored in the evaluation model pool.

[0237] For the first action decision model corresponding to the first virtual character, upon completion of the training game, the computer device determines that the action decision model has completed the i-th round of training. Subsequently, the action decision model that has completed the i-th round of training is evaluated to obtain a corresponding performance evaluation result.

[0238] Optionally, the computer device selects at least two trained control models from the evaluation model pool, and then creates at least two evaluation games, each of which includes a first virtual character that executes the action output by the action decision model, and a second virtual character that executes the action output by the control model, and then determines the performance evaluation result based on the game data of the at least two evaluation games.

[0239] The performance evaluation indicator may be the win-loss rate of the game, or the proportion of a specific behavior in the game, etc., which is not limited in this embodiment.

[0240] When the performance evaluation result indicates that the action decision model has not reached the training completion standard, the computer device needs to control the action decision model to enter the i+1th round of training, that is, when the performance evaluation result indicates that the action decision model has not reached the training completion standard, the action decision model needs to be controlled to enter the next round of training.

[0241] In the case where the performance evaluation result indicates that the action decision model has reached the training completion standard, it is determined that the action decision model has completed training.

[0242] Indicatively, the specific process is as follows:

[0243] (1) Based on the client's built-in behavior tree, the opponent virtual character is controlled and the first action decision model is trained for round 0; (2) the model trained for round 0 is stored in the evaluation model pool and evaluated; (3) based on the first action decision model that has completed round 0 training, the second action decision model is trained for round 0 and added to the opponent model pool; (4) a second decision model that has been trained in the past is selected from the opponent model pool as the opponent and the first action decision model is trained for round K; (5) the first action decision model trained for round K is stored in the evaluation model pool and evaluated; (6) the second action decision model is trained for round K based on the first action decision model trained for round K and stored in the opponent model pool; (7) steps (4) to (6) are repeated until the evaluation result of the first action decision model satisfies the performance evaluation result, and the first action decision model is determined to have completed training.

[0244] In an embodiment of the present application, by training the second action decision model corresponding to the sparring opponent virtual character and the first decision model corresponding to the first virtual character, the strength of the first decision model is improved, so that the first decision model can have stable action decision-making capabilities.

[0245] In the embodiment of the present application, whether the action decision model has completed training needs to be judged in combination with the game index and the anthropomorphism index. The game index is used to judge the strength of the action decision model, and the anthropomorphism index is used to judge the degree of anthropomorphism of the action decision model.

[0246] Please refer to Table 4, which shows the game indicators and anthropomorphic indicators provided by an exemplary embodiment of the present application.

[0247] Table 4

[0248] Among them, when determining the anthropomorphic index, the average anthropomorphic index is determined based on a large amount of game data.

[0249] After completing the i-th round of training for the action decision model, the computer device obtains game data from the i-th round of training games. Based on the game data from at least two training rounds, a game index and an anthropomorphic index are determined. The anthropomorphic index refers to the percentage of specific actions actually executed in the game data from at least two training games.

[0250] When the game metrics meet the training completion criteria and the anthropomorphic metrics match the target execution ratio of specific actions performed by real players in the game, the action decision model is determined to have completed training.

[0251] In an embodiment of the present application, game indicators and anthropomorphism indicators are jointly used as evaluation criteria for the action decision model, so that the trained action decision model can take into account both the strength and the degree of anthropomorphism of the action model.

[0252] Please refer to Figure 15, which shows a schematic diagram of the interaction between a client and a server for training an action decision model during the training process, according to an exemplary embodiment of the present application. Client 1501 is an application installed on a computer device that provides a virtual environment. Server 1502 is used to provide action model training services. During the training process, client 1501 sends status information to server 1502. After receiving the status information, server 1502 processes the status information through the Agent component in the action decision model and assembles it into input features for the action decision model. It then requests the Actor component to obtain sample target actions under the current policy. This sample target action is mapped to generate a response packet that is sent to client 1501. Furthermore, the data generated during this process undergoes reward calculation and sample processing to generate training samples, which are then sent in batches to the Learner component for policy parameter optimization. The Learner component stores the training samples received in a local buffer pool. Each training step draws from the buffer pool based on the policy. After several training steps, the Learner component synchronizes the parameters of the current training network with the target network in the Actor component. Optionally, in the embodiment of the present application, a PPO (Proximal Policy Optimization) algorithm may be used for training.

[0253] Please refer to FIG16 , which shows a schematic diagram of a decision-making mode for implementing an action decision request during a training process according to an exemplary embodiment of the present application.

[0254] During the training process, the client adopts a synchronous decision-making mode, that is, after sending the status information, it waits for the action package returned by the server and executes the sample target action based on the action package. When an action decision is needed, in a game frame, the status information is sent to the first server, and the first server sends the request data to the second server that provides the action decision service. After the action decision is made, the second server returns the response data to the first server and waits for the next frame action decision request. The first server obtains the sample target action and then returns the action instruction to the client to control the target virtual character to perform the sample target action. The use of a synchronous decision-making mode is conducive to ensuring the effect of model training.

[0255] Please refer to Figure 17, which shows a schematic diagram of a decision-making mode for implementing an action decision request in the application process of an exemplary embodiment of the present application. In the application stage, an asynchronous decision-making mode is adopted, and the client does not wait for the returned action package after sending the status information to the first server. After sending the status information, it periodically checks whether the returned action package is received while running other business logic. In the case of checking that the returned action package is not received, the business logic continues to run. In the case of checking that the returned action package is received, the target virtual character is controlled to perform the target action, and then waits for the next action decision. In the application stage, the asynchronous decision-making mode is adopted, which can effectively reduce the data blocking of the main thread, ensure that the time-consuming reasoning of the action decision model does not affect other business logics, reduce the resource usage of the business-side server (first server), and ensure the security of the action decision model.

[0256] In one possible implementation, a computer device may train an action decision model using a combination of supervised learning and reinforcement learning. In the early stages of training, the action decision model is first trained using supervised learning based on real player game data. This can effectively avoid meaningless exploration of reinforcement learning in the early stages of action decision model training. In subsequent training, reinforcement learning is then used to effectively improve the training efficiency of the action decision model.

[0257] Please refer to FIG18 , which shows a schematic structural diagram of an action decision device for a virtual character provided by an exemplary embodiment of the present application. The device includes the following structure.

[0258] An acquisition module 1801 is configured to acquire state information, wherein the state information is used to represent a game state of the first virtual character;

[0259] Decision module 1802 is configured to input the state information into an action decision model to obtain n target sub-actions serially output by n action output heads in the action decision model, wherein different action output heads correspond to different action types, and the n action output heads are serially connected based on dependency relationships between the action types, wherein the dependency relationships are used to characterize dependency constraints between sub-actions under different action types, and n is a positive integer;

[0260] The control module 1803 is configured to control the first virtual character to execute a target action consisting of the n target sub-actions.

[0261] Optionally, the action decision model includes an information processing layer, a feature extraction layer, and an action decision layer, and the action decision layer includes n action output heads;

[0262] The decision module 1802 is used to:

[0263] Inputting the state information into the action decision model, encoding the state information through the information processing layer to obtain a state code;

[0264] Inputting the state code into the feature extraction layer, extracting the state code features through the feature extraction layer to obtain fusion features;

[0265] Based on the fusion features, n target sub-actions are determined by the n action output heads in the action decision layer.

[0266] Optionally, the decision module 1802 is configured to:

[0267] Inputting the fused feature into a first action output head in the action decision layer, and determining a first target sub-action from a first action type through the first action output head;

[0268] The embedded coding vectors corresponding to the determined first target sub-action to the i-1th target sub-action, and the fusion features are input into the action decision layer, and the i-th target sub-action is determined from the i-th action type through the i-th action output head in the action decision layer, wherein the first target sub-action to the i-1th target sub-action are respectively determined by the first action output head to the i-1th action output head, i is less than or equal to n, and i is greater than 1.

[0269] Optionally, the action decision model further includes an action mask layer, the action mask layer is connected to the n action output heads in the action decision layer, and the action mask layer is used to mask sub-actions in different action types;

[0270] The device further comprises:

[0271] an action masking module, configured to mask the sub-actions in the i-th action type based on the determined dependency restriction relationship indicated by the dependency restriction conditions between the first target sub-action to the i-1-th target sub-action and sub-actions in different action types, wherein at least two sub-actions having the dependency restriction relationship cannot be executed simultaneously;

[0272] The decision module 1802 is configured to determine the i-th target sub-action from unmasked sub-actions in the i-th action type through the i-th action output head in the action decision layer.

[0273] Optionally, the decision module 1802 is configured to:

[0274] If there is a determined j-th target sub-action, and the dependency restriction condition indicates that the j-th target sub-action has the dependency restriction relationship with at least one first sub-action in the i-th action type, masking the first sub-action in the i-th action type, where j is less than i;

[0275] or,

[0276] When there is a determined x-th target sub-action and y-th target sub-action, and the dependency restriction situation indicates that after the x-th target sub-action is combined with the y-th target sub-action, the dependency restriction relationship exists with at least one second sub-action in the i-th action type, mask the second sub-action in the i-th action type, and x, y is less than i.

[0277] Optionally, the state information includes character state information of the first virtual character in the game, and environmental perception information of the virtual environment in which the first virtual character is located. The character state information is used to represent the interaction state between the first virtual character and the second virtual character in the game, and the interaction state between the first virtual character and the virtual environment in the game. The character state information is a one-dimensional vector, and the environmental perception information is a two-dimensional image. The information processing layer includes a scalar encoder and an image encoder.

[0278] Optionally, the decision module 1802 includes:

[0279] Inputting the state information into the action decision model, encoding the character state information through the scalar encoder in the information processing layer to obtain a state information encoding result;

[0280] Encoding the environmental perception information by the image encoder in the information processing layer to obtain an environmental information encoding result;

[0281] The state information encoding result and the environment information encoding result are spliced ​​together by the information processing layer to obtain the state code.

[0282] Optionally, the action decision model further includes an information filtering layer, wherein the information filtering layer is connected to the output end of the scalar encoder and the information filtering layer is connected to the input end of the action decision layer;

[0283] The device further comprises:

[0284] A filtering module is used to filter the character status information through the information filtering layer when the character status information is obtained to obtain filtered character status information, wherein the filtered character status information has a correlation with the first action type corresponding to the first action output head; and input the filtered target character status information into the first action output head.

[0285] Optionally, the device further includes:

[0286] The decomposition module is used to orthogonally decompose the executable actions of the first virtual character to obtain n action types, where different action types include at least two executable sub-actions, and the sub-actions under different action types support independent control.

[0287] Optionally, the device further includes:

[0288] A training module, configured to obtain sample state information, wherein the sample state information is used to represent a game state of a training game in which the first virtual character is playing;

[0289] The training module is further used to train the action decision model through reinforcement learning based on the sample state information.

[0290] Optionally, the training module is used to:

[0291] Inputting the sample state information into the action decision model to obtain a sample target action output by the action decision model, wherein the sample target action is composed of n sample target sub-actions;

[0292] controlling the first virtual character to execute the sample target action, and obtaining a sample action execution result of the first virtual character;

[0293] determining an action execution reward based on the sample action execution result of the first virtual character;

[0294] Model parameters of the action decision model are updated based on the action execution reward.

[0295] Optionally, the information processing layer further includes a game information encoder, and the sample state information further includes sample game information, where the sample game information is used to represent the real-time situation of the training game in which the first virtual character is playing;

[0296] The training module is configured to encode the sample game information by the game information encoder to obtain a game state code when the sample state information is input into the action decision model;

[0297] The training module is further used to determine the action execution reward based on the game state code and the sample action execution result.

[0298] Optionally, the action execution reward includes at least one of an attribute reward, a game win reward, a game lose reward, and a task reward, wherein the game win reward is a positive reward and the game lose reward is a negative reward;

[0299] The training module is used to:

[0300] When the first virtual character wins the game, determining the game victory reward as the action execution reward;

[0301] In the case where the first virtual character fails in the game, determining the game failure reward as the action execution reward;

[0302] In a case where the attribute value of the first virtual character decreases, determining the attribute reward as the action execution reward;

[0303] When the task of the first virtual character is successful, the task reward is determined as the action execution reward.

[0304] Optionally, the action execution reward includes an anthropomorphic reward;

[0305] The training module is used to:

[0306] determining anthropomorphic attribute values ​​of at least two sample target actions based on sample action execution results of the first virtual character performing at least two sample target actions in the training game;

[0307] In a case where the anthropomorphic attribute value is lower than the anthropomorphic attribute threshold, the anthropomorphic reward is determined as the action execution reward.

[0308] Optionally, the training module is used to:

[0309] When the training game is finished, determining that the action decision model has completed the i-th round of training;

[0310] Evaluate the action decision model after completing the i-th round of training to obtain a performance evaluation result;

[0311] If the performance evaluation result indicates that the action decision model has not reached the training completion standard, controlling the action decision model to enter the (i+1) round of training;

[0312] In a case where the performance evaluation result indicates that the action decision model has reached a training completion standard, it is determined that the action decision model has completed training.

[0313] Optionally, there is at least one second virtual character in the training game in which the first virtual character is located, and the second virtual character executes an action output by the second action decision model;

[0314] The training module is configured to, when the first action decision model corresponding to the first virtual character completes the i-th round of training, control the second action decision model to enter the i-th round of training based on the trained first action decision model, and store the second action decision model that has completed the i-th round of training in the evaluation model pool;

[0315] The training module is further used to:

[0316] Selecting at least two trained control models from the evaluation model pool;

[0317] Creating at least two evaluation games, wherein the evaluation games include the first virtual character executing the action output by the action decision model and a second virtual character executing the action output by the comparison model;

[0318] The performance evaluation result is determined based on the game data of the at least two evaluation games.

[0319] Optionally, the training module is used to:

[0320] Upon completion of the i-th round of training of the action decision model, obtaining game data of the training game in which the i-th round of training is performed;

[0321] Determining a game index and an anthropomorphic index based on the game data of the training games of at least two rounds of training, wherein the anthropomorphic index refers to a percentage of specific actions actually executed in the at least two rounds of training games;

[0322] When the game indicator meets the training completion standard and the actual execution ratio indicated by the anthropomorphic indicator matches the target execution ratio, it is determined that the action decision model has completed training. The target execution ratio is the ratio of real players executing the feature action in real games.

[0323] In summary, in the embodiments of the present application, action decisions are made based on state information through an action decision model, thereby being able to determine a reasonable target action based on the current state. The action decision model includes n action output heads connected in series, each of which is used to output different target sub-actions. The n action output heads are connected in series based on the dependencies between action types. This allows the dependencies between different actions to be considered when making action decisions, and in the process of serially outputting the n target sub-actions, the n target sub-actions are causally related in time sequence, thereby improving the rationality of the target action and enhancing the anthropomorphism of the virtual character.

[0324] Please refer to Figure 19, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device can be implemented as a terminal or server in the above-mentioned embodiment. Specifically, the computer device 1900 includes a central processing unit (CPU) 1901, a system memory 1904 including a random access memory 1902 and a read-only memory 1903, and a system bus 1905 connecting the system memory 1904 and the central processing unit 1901. The computer device 1900 also includes a basic input / output system (I / O system) 1906 that helps transmit information between various components in the computer, and a large-capacity storage device 1907 for storing an operating system 1913, application programs 1914, and other program modules 1915.

[0325] In some embodiments, the basic input / output system 1906 includes a display 1908 for displaying information and an input device 1909, such as a mouse or keyboard, for user input. Both the display 1908 and the input device 1909 are connected to the central processing unit 1901 via an input / output controller 1910 connected to the system bus 1905. The basic input / output system 1906 may also include an input / output controller 1910 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1910 also provides output to a display screen, printer, or other types of output devices.

[0326] The mass storage device 1907 is connected to the central processing unit 1901 via a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 and its associated computer-readable media provide non-volatile storage for the computer device 1900. In other words, the mass storage device 1907 may include computer-readable media (not shown) such as a hard disk or drive.

[0327] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1904 and mass storage device 1907 can be collectively referred to as memory.

[0328] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1901. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1901 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0329] According to various embodiments of the present application, the computer device 1900 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1900 may be connected to a network 1912 via a network interface unit 1911 connected to the system bus 1905. Alternatively, the network interface unit 1911 may be used to connect to other types of networks or remote computer systems (not shown).

[0330] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0331] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer instruction. The at least one computer instruction is loaded and executed by a processor to implement the action decision method of the virtual character described in any of the above embodiments.

[0332] The present invention provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to execute the action decision method for a virtual character provided in the above aspects.

[0333] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be performed by a program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, which can be the computer-readable storage medium included in the memory in the above embodiments, or a separate computer-readable storage medium not incorporated into the terminal. The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the action decision method for the virtual character described in any of the above method embodiments.

[0334] Optionally, the computer-readable storage medium may include: ROM, RAM, solid-state drives (SSDs), or optical disks. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments.

[0335] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0336] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0337] Before collecting the user's relevant data and during the process of collecting the user's relevant data, this application can display a prompt interface, pop-up window or output voice prompt information. The prompt interface, pop-up window or voice prompt information is used to remind the user that his or her relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the user's relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining the user's relevant data are terminated, that is, the user's relevant data is not obtained.

[0338] It should be understood that the "plurality" mentioned in this article refers to two or more. The "first", "second", etc. mentioned in this article are used to distinguish similar objects, and are not used to limit a specific order or sequence. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may also be executed in a non-numbered order, such as two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in an order opposite to that shown in the figure. This embodiment of the application does not limit this.

[0339] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for action decision of a virtual character, the method being executed by a computer device, the method comprising: Acquiring status information, where the status information is used to represent a game status of the game in which the first virtual character is playing; Inputting the state information into an action decision model, obtaining n target sub-actions serially outputted by n action output heads in the action decision model, wherein different action output heads correspond to different action types, and the n action output heads are serially connected based on dependency relationships between the action types, wherein the dependency relationships are used to characterize dependency restrictions between sub-actions under different action types, and n is a positive integer; The first virtual character is controlled to execute a target action consisting of the n target sub-actions.

2. The method according to claim 1, wherein: The action decision model includes an information processing layer, a feature extraction layer and an action decision layer, and the action decision layer includes n action output heads; The step of inputting the state information into the action decision model to obtain n target sub-actions serially outputted by n action output heads in the action decision model comprises: Inputting the state information into the action decision model, encoding the state information through the information processing layer to obtain a state code; Inputting the state code into the feature extraction layer, extracting the state code feature through the feature extraction layer to obtain a fusion feature; Based on the fusion features, n target sub-actions are determined through the n action output heads in the action decision layer.

3. The method according to claim 2, wherein: The determining the n target sub-actions through the n action output heads in the action decision layer based on the fusion features includes: Inputting the fused feature into a first action output head in the action decision layer, and determining a first target sub-action from a first action type through the first action output head; The embedded coding vectors corresponding to the determined first target sub-action to the i-1th target sub-action and the fusion features are input into the action decision layer, and the i-th target sub-action is determined from the i-th action type through the i-th action output head in the action decision layer, wherein the first target sub-action to the i-1th target sub-action are respectively determined by the first action output head to the i-1th action output head, i is less than or equal to n, and i is greater than 1.

4. The method according to claim 3, wherein: The action decision model further comprises an action mask layer, the action mask layer is connected to the n action output heads in the action decision layer, and the action mask layer is used to mask sub-actions in different action types; The method further comprises: Based on the determined first target sub-action to the (i-1)th target sub-action, and the dependency restriction relationship indicated by the dependency restriction situations between sub-actions under different action types, masking the sub-actions in the (i)th action type, wherein at least two sub-actions having the dependency restriction relationship do not support simultaneous execution; The determining the i-th target sub-action from the i-th action type through the i-th action output head in the action decision layer includes: The i-th target sub-action is determined from the unmasked sub-actions in the i-th action type through the i-th action output head in the action decision layer.

5. The method according to claim 4, wherein: The step of masking the sub-actions in the i-th action type based on the determined first target sub-action to the i-1th target sub-action and the dependency restriction relationship indicated by the dependency restriction conditions between sub-actions in different action types includes: When there is a determined j-th target sub-action, and the dependency constraint indicates that the j-th target sub-action is related to the When at least one first sub-action in the i-th action type exists in the dependency restriction relationship, masking the first sub-action in the i-th action type, j is less than i; or, When there are determined x-th target sub-action and y-th target sub-action, and the dependency restriction situation indicates that after the x-th target sub-action is combined with the y-th target sub-action, it has the dependency restriction relationship with at least one second sub-action in the i-th action type, mask the second sub-action in the i-th action type, and x, y is less than i.

6. The method according to any one of claims 2 to 5, wherein: The state information includes character state information of the first virtual character in the game, and environmental perception information of a virtual environment in which the first virtual character is located, the character state information is used to represent an interaction state between the first virtual character and a second virtual character in the game, and an interaction state between the first virtual character and the virtual environment in the game, the character state information is a one-dimensional vector, the environmental perception information is a two-dimensional image, and the information processing layer includes a scalar encoder and an image encoder; The step of inputting the state information into the action decision model and encoding the state information through the information processing layer to obtain the state code comprises: Inputting the state information into the action decision model, encoding the character state information through the scalar encoder in the information processing layer, and obtaining a state information encoding result; Encoding the environment perception information by the image encoder in the information processing layer to obtain an environment information encoding result; The state information encoding result and the environment information encoding result are concatenated by the information processing layer to obtain the state encoding.

7. The method according to claim 6, wherein: The action decision model further includes an information filtering layer, the information filtering layer is connected to the output end of the scalar encoder, and the information filtering layer is connected to the input end of the action decision layer; The method further comprises: When the character status information is obtained, filtering the character status information through the information filtering layer to obtain filtered character status information, wherein the filtered character status information has a correlation with the first action type corresponding to the first action output head; The filtered target character state information is input into the first action output head.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: The executable actions of the first virtual character are orthogonally decomposed to obtain n action types, where different action types include at least two executable sub-actions, and the sub-actions under different action types support independent control.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: Acquire sample status information, where the sample status information is used to represent a game status of a training game in which the first virtual character is in; Based on the sample state information, the action decision model is trained by reinforcement learning.

10. The method according to claim 9, wherein: The step of training the action decision model by reinforcement learning based on the sample state information includes: Inputting the sample state information into the action decision model to obtain a sample target action output by the action decision model, wherein the sample target action is composed of n sample target sub-actions; Controlling the first virtual character to execute the sample target action, and obtaining a sample action execution result of the first virtual character; Determining an action execution reward based on the sample action execution result of the first virtual character; Model parameters of the action decision model are updated based on the action execution reward.

11. The method according to claim 10, wherein: The information processing layer further includes a game information encoder, and the sample state information further includes sample game information, where the sample game information is used to represent the real-time situation of the training game in which the first virtual character is in; The method further comprises: When the sample state information is input into the action decision model, the sample game information is encoded by the game information encoder to obtain a game state code; The determining of an action execution reward based on the sample action execution result of the first virtual character includes: The action execution reward is determined based on the game status code and the sample action execution result.

12. The method according to claim 10, wherein: The action execution reward includes at least one of an attribute reward, a game victory reward, a game failure reward, and a task reward, wherein the game victory reward is a positive reward, and the game failure reward is a negative reward; The determining of an action execution reward based on the sample action execution result of the first virtual character includes: When the first virtual character wins the game, determining the game victory reward as the action execution reward; In the case where the first virtual character fails in the game, determining the game failure reward as the action execution reward; In a case where the attribute value of the first virtual character decreases, determining the attribute reward as the action execution reward; When the task of the first virtual character is successful, the task reward is determined as the action execution reward.

13. The method according to claim 11, wherein: The action execution reward includes an anthropomorphic reward; The determining of an action execution reward based on the sample action execution result of the first virtual character includes: Determining anthropomorphic attribute values ​​of at least two sample target actions based on sample action execution results of the first virtual character performing at least two sample target actions in the training game; In a case where the anthropomorphic attribute value is lower than the anthropomorphic attribute threshold, the anthropomorphic reward is determined as the action execution reward.

14. The method according to claim 9, wherein: The method further comprises: When the training game is finished, determining that the action decision model has completed the i-th round of training; Evaluate the action decision model that has completed the i-th round of training to obtain a performance evaluation result; When the performance evaluation result indicates that the action decision model has not reached the training completion standard, controlling the action decision model to enter the (i+1)th round of training; In a case where the performance evaluation result indicates that the action decision model has reached a training completion standard, it is determined that the action decision model has completed training.

15. The method according to claim 14, wherein: There is at least one second virtual character in the training game in which the first virtual character is located, and the second virtual character performs an action output by a second action decision model; The method further comprises: When the first action decision model corresponding to the first virtual character completes the i-th round of training, based on the trained first action decision model, controlling the second action decision model to enter the i-th round of training, and storing the second action decision model that has completed the i-th round of training in the evaluation model pool; The evaluation completes the action decision model of the i-th round of training to obtain a performance evaluation result, including: Selecting at least two trained control models from the evaluation model pool; Creating at least two evaluation games, wherein the evaluation games include the first virtual character executing the action output by the action decision model and a second virtual character executing the action output by the comparison model; The performance evaluation result is determined based on the game data of the at least two evaluation games.

16. The method according to claim 9, wherein: The method further comprises: Upon completion of the i-th round of training of the action decision model, obtaining game data of the training game in which the i-th round of training is performed; Determine a game index and an anthropomorphic index based on the game data of the training games of at least two rounds of training, wherein the anthropomorphic index refers to the actual execution ratio of a specific action in the at least two rounds of the training games; When the game indicator reaches the training completion standard and the actual execution ratio indicated by the anthropomorphic indicator matches the target execution ratio, it is determined that the action decision model has completed training. The target execution ratio is the ratio of real players executing the feature action in real games.

17. A virtual character action decision device, the device comprising: An acquisition module, used for acquiring status information, wherein the status information is used for representing the game status of the first virtual character; A decision module, used for inputting the state information into an action decision model, and obtaining n target sub-actions serially outputted by n action output heads in the action decision model, wherein different action output heads correspond to different action types, and the n action output heads are serially connected based on the dependency relationship between the action types, and the dependency relationship is used to characterize the dependency restriction between sub-actions under different action types, and n is a positive integer; A control module is used to control the first virtual character to execute a target action consisting of the n target sub-actions.

18. A computer device, comprising a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the action decision method of a virtual character as described in any one of claims 1 to 16.

19. A computer-readable storage medium, wherein at least one computer instruction is stored in the computer-readable storage medium, and the at least one computer instruction is loaded and executed by a processor to implement the action decision method of a virtual character as described in any one of claims 1 to 16.

20. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the action decision method for a virtual character as described in any one of claims 1 to 16.

Images