Action decision-making method and apparatus for virtual character, and device and storage medium

Abstract

An action decision-making method and apparatus for a virtual character, and a device and a storage medium. The method comprises: acquiring character state information of a first virtual character, and environment perception information of the first virtual character with regard to a virtual environment where the first virtual character is currently located; fusing coding results corresponding to the character state information and the environment perception information, respectively, so as to obtain a fused feature; on the basis of the fused feature, performing action decision-making on the first virtual character, so as to obtain a character action; and controlling the first virtual character to execute the character action.

Description

Virtual character action decision method, device, equipment and storage medium

[0001] This application claims priority to Chinese patent application No. 202311198980.6 filed on September 15, 2023, entitled “Virtual Character Action Decision Method, Device, Equipment and Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for making action decisions for a virtual character. Background Art

[0003] Nowadays, when planning video games, people usually hope that the behavior logic of NPCs (Non-Player Characters) can be as consistent as possible with the behavior logic of real players, that is, the NPCs in the game have highly anthropomorphic characteristics.

[0004] In related technologies, when determining the behavioral logic of a virtual character based on a neural network model, the virtual character's behavior is typically predicted based on the virtual character's attributes in the virtual environment and virtual environment information. This can be done by performing image recognition on the original pixel image of the virtual environment to determine the virtual environment information of the virtual character. This virtual environment information is then input into the neural network model, and the virtual character is controlled to perform the actions output by the neural network model.

[0005] However, some virtual environments are large in scale and the scene content is relatively complex. For example, there are various moving obstacles in the virtual environment, such as airplanes, cars, and virtual characters. If the virtual environment information is determined based on image recognition of the current virtual environment image, more resources will be consumed when collecting the virtual environment image, which makes the determination of the virtual character action more complex and the labor cost higher.

[0006] Summary of the Invention

[0007] The present invention provides a method, apparatus, device, and storage medium for making action decisions for a virtual character. The technical solution is as follows:

[0008] In one aspect, an embodiment of the present application provides a method for determining an action of a virtual character, which is executed by a computer device. The method includes:

[0009] Obtaining character state information of the first virtual character, and obtaining environmental perception information of the first virtual character regarding the virtual environment in which the character is currently located, wherein the environmental perception information includes perception information of multiple modalities, each of the multiple modalities representing an information dimension;

[0010] Fusing the encoding results corresponding to the character state information and the environmental perception information to obtain a fusion feature, wherein the encoding result is a result obtained by performing feature encoding on the character state information and multiple perception information respectively using different encoding methods;

[0011] Performing action decision on the first virtual character based on the fusion feature to obtain a character action;

[0012] Controlling the first virtual character to perform the character action.

[0013] On the other hand, this embodiment provides a virtual character action decision device, the device comprising:

[0014] an acquisition module, configured to acquire character state information of a first virtual character and environmental perception information of a current virtual environment of the first virtual character, wherein the environmental perception information includes perception information of multiple modalities, each of the multiple modalities representing an information dimension;

[0015] an encoding module, configured to fuse encoding results corresponding to the character state information and the environmental perception information to obtain a fusion feature, wherein the encoding result is a result of feature encoding the character state information and the plurality of perception information using different encoding methods;

[0016] a decision module, configured to make an action decision for the first virtual character based on the fusion feature to obtain a character action;

[0017] A control module is used to control the first virtual character to perform the character action.

[0018] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the action decision method of the virtual character as described in the above aspects.

[0019] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the action decision method of the virtual character as described in the above aspects.

[0020] In another aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the action decision-making method for a virtual character provided in the above aspects.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0022] In an embodiment of the present application, when an action decision needs to be made for a first virtual character in a virtual environment, the character state information and multimodal environmental perception information of the first virtual character are obtained, and feature encoding and fusion of the character state information and multimodal environmental perception information are performed, and finally an action decision is made based on the fused features. When making an action decision, obtaining environmental perception information of different information dimensions perceived by the first virtual character helps to obtain a more comprehensive and detailed three-dimensional virtual environment around the first virtual character, thereby enabling more reasonable action decisions. Using different encoding methods to feature encode the character state information and multiple different modal perception information can make the action decision model fit the various types of information in the current scene more accurately. In addition, compared with the method of obtaining environmental features by image recognition in related technologies, by extracting features from quantified multimodal environmental perception information and then fusing the encoding results of the character state information, it helps to more fully consider the character state information of the first virtual character itself while quantifying the environmental perception information, while saving computing resources and improving the accuracy of the character action decision, making the effect of the first virtual character performing the role action more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application;

[0024] FIG2 shows a flow chart of a method for making action decisions for a virtual character provided by an exemplary embodiment of the present application;

[0025] FIG3 shows a schematic diagram of a ray detection solution provided by an exemplary embodiment of the present application;

[0026] FIG4 shows a schematic diagram of second modality perception information provided by an exemplary embodiment of the present application;

[0027] FIG5 shows a schematic diagram of a grid structure provided by an exemplary embodiment of the present application;

[0028] FIG6 shows a flowchart of a process for generating character actions provided by an exemplary embodiment of the present application;

[0029] FIG7 is a schematic diagram showing a process of performing one-dimensional convolution on environmental perception information provided by an exemplary embodiment of the present application;

[0030] FIG8 shows a schematic diagram of a user interface of an application for providing a virtual environment provided by an exemplary embodiment of the present application;

[0031] FIG9 shows a schematic diagram of a moving direction and sub-action components describing an action provided by an exemplary embodiment of the present application;

[0032] FIG10 shows a schematic diagram of an autoregressive structure of n action output heads provided by an exemplary embodiment of the present application;

[0033] FIG11 is a schematic diagram showing visual situation perception provided by an exemplary embodiment of the present application;

[0034] FIG12 is a schematic diagram showing a method for determining an effective aiming range in a vertical direction according to an exemplary embodiment of the present application;

[0035] FIG13 is a schematic diagram showing an effective aiming range in the horizontal direction provided by an exemplary embodiment of the present application;

[0036] FIG14 shows a schematic diagram of an action decision model provided by an exemplary embodiment of the present application;

[0037] FIG15 shows a flowchart of an action decision model training process provided by an exemplary embodiment of the present application;

[0038] FIG16 is a schematic diagram showing a method of dividing a depth map into regions according to an exemplary embodiment of the present application;

[0039] FIG17 is a schematic diagram showing a turning sub-action in a character action provided by an exemplary embodiment of the present application;

[0040] FIG18 shows a schematic structural diagram of an action decision-making device for a virtual character provided by an exemplary embodiment of the present application;

[0041] FIG19 shows a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0043] The embodiments of the present application relate to artificial intelligence (AI) and machine learning technology, and are designed based on machine learning (ML) in artificial intelligence.

[0044] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technologies such as reinforcement learning and deep learning have widespread applications in various fields. For illustrative purposes, this application relates to reinforcement learning technology in machine learning.

[0045] Reinforcement learning (RL) is a branch of machine learning in which an agent learns by interacting with its environment. This is a result-oriented learning process in which the agent is not told what actions to take; instead, the agent learns from the results of its actions.

[0046] The intelligent agent can perceive the environment through sensors and interact with the environment through actuators. For each possible perception sequence, the intelligent agent should select an action to perform that can achieve the expected maximum performance under the conditions that it has instructions and evidence provided by the perception sequence. In the embodiment of the present application, the first virtual character acts as an intelligent agent to perceive the virtual environment, obtain multimodal environmental perception information, and execute the character action determined by the action decision model.

[0047] A virtual environment is a virtual environment displayed (or provided) when an application is running on a computer device. This virtual environment can be a simulation of the real world, a semi-simulated and semi-imaginary environment, or a purely imaginary environment. This application does not limit this to any of the following: a two-dimensional virtual environment, a 2.5-dimensional virtual environment, or a three-dimensional virtual environment. The following embodiments illustrate a three-dimensional virtual environment.

[0048] A virtual character refers to an movable object in a virtual environment. The movable object can be at least one of a virtual character, a virtual animal, and an animated character. Optionally, when the virtual environment is a three-dimensional virtual environment, the virtual character can be a three-dimensional virtual model. Each virtual character has its own shape and volume in the three-dimensional virtual environment and occupies a portion of the space in the three-dimensional virtual environment. Optionally, the virtual character is a three-dimensional character constructed based on three-dimensional human skeleton technology, and the virtual character achieves different external appearances by wearing different skins. In some implementations, the virtual character can also be implemented using a 2.5-dimensional or 2-dimensional model, which is not limited in the embodiments of the present application.

[0049] Related technologies use raw pixels as a means of environmental perception and perform image recognition based on raw pixel information. This requires collecting image data from the client and then performing further image recognition to determine the state of the virtual environment. However, some virtual environments are large and complex, such as those containing various moving obstacles, such as airplanes, cars, and virtual characters. If this information is determined based on image recognition of the current virtual environment, the acquisition of the virtual environment image consumes a large amount of resources, making the determination of virtual character movements more complex and labor-intensive.

[0050] In addition, in the related art, multiple rays may be emitted in the navigation grid where the virtual character is located. The rays stop when they hit the edge of the navigation network, and then a number of waypoints are screened out based on the scene query system to ensure the passability of the virtual character. However, the method of querying discrete points only provides the first virtual character with a number of waypoints to reach the destination, and cannot clearly express the characteristics of the surrounding environment, which is not conducive to the deep neural network to obtain the surrounding environment characteristics. And when the frame rate of the deep neural network for action decision-making is high, the virtual character will pull and swing between different waypoints, and the virtual character will twitch. Therefore, an embodiment of the present application provides a method for action decision-making of a virtual character, which can obtain more refined environmental features by obtaining the multi-modal environmental perception features perceived by the first virtual character, thereby realizing the fusion of the surrounding environment and the state features of the first virtual character to obtain fusion features. Action decisions are then made based on the fusion features to control the actions of the first virtual character.

[0051] The computer device in this application can be a desktop computer, a laptop computer, a mobile phone, a tablet computer, a desktop computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, an intelligent voice interaction device, a smart home appliance, and a vehicle-mounted terminal, etc. An application that supports a virtual environment, such as an application that supports a three-dimensional virtual environment, is installed and run in the computer device. The application can be any one of a virtual reality application, a three-dimensional map program, a TPS (Third-Person Shooter) game, an FPS (First-Person Shooter) game, and a MOBA (Multiplayer Online Battle Arena) game. Optionally, the application can be a stand-alone application, such as a stand-alone 3D game program, or a network-connected application. The following embodiments are illustrated by using applications in games.

[0052] Games based on virtual environments are often composed of one or more game world maps. The virtual environment in the game simulates scenes in the real world. The first virtual character can walk, run, jump, shoot, fight, drive, climb, glide, switch to using virtual props, use virtual props to attack other virtual characters, etc. in the virtual environment.

[0053] Please refer to FIG1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment may include: a terminal 110 , a first server 120 , and a second server 130 .

[0054] In the embodiment of the present application, terminal 110 includes, but is not limited to, mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals. Terminal 110 runs an application 111 that supports a virtual environment. This application can be a TPS (Third-Person Shooter) game or an FPS (First-Person Shooter) game. When terminal 110 runs application 111, the user interface of application 111 is displayed on the screen of terminal 110. A user uses terminal 110 to control a virtual character in the virtual environment to perform activities, or the terminal controls an NPC (Non-Player Character) in the virtual environment to perform activities. The activities of the virtual character include, but are not limited to, adjusting body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, throwing, and performing skills.

[0055] The first server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In an embodiment of the present application, the first server 120 is used to provide background services for applications that support three-dimensional virtual environments, and can be a DS (Dedicated Server). Optionally, the first server 120 undertakes the main computing work and the terminal undertakes the secondary computing work; or, the first server 120 undertakes the secondary computing work and the terminal undertakes the main computing work; or, a distributed computing architecture is used between the first server 120 and the terminal for collaborative computing.

[0056] The second server 130 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In an embodiment of the present application, the second server 130 is used to support the application of the three-dimensional virtual environment to provide a virtual character action decision service. When the action decision service provided by the second server 130 is turned on, its own address is registered in ETCD (et-see-dee, distributed key-value pair storage). In the case of an action decision requirement for a virtual character, the first server 120 requests the service dispatcher to connect to the inference service and obtains the address of the action decision service returned by the service dispatcher. Thus, the first server 120 sends an action decision request to the action decision service provided by the second server 130 through the obtained address. After the inference service determines the character action, it returns the action instruction to the first server 120.

[0057] In some embodiments, a method for executing an action decision of a virtual character is performed through an action decision model. During the process of training the action decision model, the second server 130 trains the action decision model through reinforcement learning based on the character state information in the action decision request sent by the first server 120 and the multimodal environmental perception information.

[0058] The embodiments of the present application can be applied to a first virtual character's navigation scenario in a virtual environment, a scenario where the character's actions complete a set task, or a scenario where the first virtual character plays against other virtual characters, and the present embodiment is not limited thereto. The following is a schematic illustration of the application process of the embodiments of the present application in a virtual scenario.

[0059] The virtual character action decision method provided in the embodiment of the present application is applied to a game scene. When an action decision is to be made for the first virtual character in the game scene, the character state information of the first virtual character and the multimodal environmental perception information of the first virtual character of the current virtual environment are obtained through the terminal 110 or the first server 120, and the character action to be performed by the first virtual character is determined based on the character state information and the environmental perception information through the second server 130. If the virtual character action decision method is executed by an action decision model, then during the training process of the action decision model, sample character state information and sample environmental perception information of the first virtual character are obtained, and the sample character state information and sample environmental perception information are input into the action decision module to train the action decision model.

[0060] It should be noted that the embodiments of the present application can also be applied to a variety of artificial intelligence scenarios. The above implementation environment is only an illustrative example and does not limit the application scenarios of the embodiments of the present application. For ease of description, the following embodiments are described using the action decision method of a virtual character in a virtual scene executed by a computer device as an example.

[0061] Please refer to Figure 2, which shows a flowchart of a method for action decision-making of a virtual character provided by an exemplary embodiment of the present application. This embodiment uses the method applied to a computer as an example to illustrate, and the method includes the following steps:

[0062] Step 201: Acquire character status information of a first virtual character and environmental perception information of the first virtual character to the current virtual environment.

[0063] Illustratively, the first virtual character is any virtual character in the virtual environment. Optionally, the first virtual character is a master virtual character controlled by the player, such as a virtual hero or a virtual mount; or the first virtual character is an automatic virtual character automatically controlled by the system, such as a virtual character automatically controlled when the player is idle.

[0064] Environmental perception information includes multiple modalities of perception information, each representing an information dimension. Therefore, environmental perception information can also be referred to as multimodal environmental perception information, where different modalities of perception information have different information dimensions.

[0065] In a game scene, a computer device extracts useful information from the game scene, including at least character state information corresponding to a first virtual character and environmental perception information. The character state information refers to the first virtual character's current attribute information, interaction information generated by the first virtual character interacting with other virtual characters, and interaction information between the first virtual character and the environment. This information can reflect the current state of the first virtual character.

[0066] The character status information is a one-dimensional vector, that is, a vector feature. In the game scene, the character status information can be directly obtained from the game engine interface. In addition, some character status information can also be obtained through ray detection.

[0067] In a game scenario, the number of frames per second a game runs is called a game frame. During each game frame, the computer device will obtain the current character status information and the current environment perception information, or the computer device will obtain the current character status information and the current environment perception information at a certain periodic interval. The larger the game frame rate, the smoother the interface display. The game frame rate can be 36, and the computer device will obtain 36 frames of character status information and the current environment perception information per second. In the case of a larger frame rate, such as 72 frames, the performance requirements for the computer device are higher. In this case, the period can be set to 3, that is, one frame of information is obtained every three game frames, and 24 frames of character status information and the current environment perception information are obtained per second.

[0068] For illustration, please refer to Table 1, which shows the classification of role status information provided by an exemplary embodiment of the present application.

[0069] Table 1

[0070] Environmental perception information may be acquired differently for different modalities, and the action decision model also uses different methods to acquire different modalities. Environmental perception information is used to obtain more comprehensive and three-dimensional virtual environment features. Optionally, the computer device can acquire the virtual character's character state information and environmental perception information once for each frame of the game screen. Alternatively, the computer device can acquire the character state information and environmental perception information once after every N frames of the game screen have been displayed. A faster frequency of feature acquisition allows for richer information to be obtained during action prediction, making the character action determination more accurate. A slower frequency of feature acquisition prevents the first virtual character from receiving a new character action before completing the first character action, thus interrupting the execution of the first character action and potentially causing errors in controlling the virtual character's action execution. For example, the computer device can acquire the character state information and environmental perception information for each frame. If the action decision process is specified using the action decision model, the character state information and environmental perception information acquired in the fourth frame can be input into the action decision model.

[0071] Step 202: The encoding results corresponding to the character state information and the environment perception information are fused to obtain a fusion feature.

[0072] Illustratively, the encoding results corresponding to the feature extraction of the character state information and the encoding results corresponding to the feature extraction of the environmental perception information are fused to obtain a fused feature. The encoding results are obtained by encoding the character state information and the plurality of perception information using different encoding methods.

[0073] Optionally, a feature extraction process is performed through an encoding network (or encoder), and the character state information and environmental perception information are input into the same encoding network, but different encoding methods are used to analyze the character state information and environmental perception information respectively, to obtain encoding results corresponding to the character state information and environmental perception information, respectively. The encoding method includes at least one of a plurality of methods such as convolutional coding, cyclic coding, autoencoder coding, and attention coding. Different encoding methods include the application of the above encoding methods alone or in combination.

[0074] Optionally, a feature extraction process is performed via an encoding network, and the character state information and environmental perception information are respectively input into different encoding networks using different encoding schemes, thereby obtaining encoding results corresponding to the character state information and environmental perception information, respectively. Optionally, considering that environmental perception information includes perceptual information from multiple modalities, each modality represents an information dimension, which is used to represent the dimension of measuring information. Illustratively, different information sources correspond to different information dimensions; or, different information acquisition methods correspond to different information dimensions, etc. Therefore, when encoding the environmental perception information, the perceptual information from multiple modalities is encoded separately. For example, during the feature extraction process via the encoding network, the character state information and the perceptual information from multiple modalities are respectively input into different encoding networks using different encoding schemes, and the outputs are encoding results corresponding to the character state information and encoding results corresponding to the perceptual information from multiple modalities. The encoding results of the perceptual information from multiple modalities can be collectively referred to as the encoding results of the environmental perception information.

[0075] In some embodiments, the action decision process is performed using an action decision model, which includes an encoding network and a decision network. The encoding network is used to encode the input information, while the decision network is used to determine the character's action. Because different modalities of perceptual information have different information dimensions, the encoding network should include convolution kernels of different dimensions, so that the perceptual information of different modalities can be encoded separately through convolution kernels of different dimensions.

[0076] Machine learning algorithms all perform linear algebra calculations on matrices, so the features involved in the calculations must be numerical. However, the input character state information and multimodal environmental perception information contain non-numeric information, so this non-numeric information needs to be encoded. For example, the character states in the table above can be converted into numerical features using label encoding or one-hot encoding.

[0077] After encoding the character state information and environmental perception information separately, two encoding results are obtained. After feature concatenation of these two encoding results, feature extraction is performed using a convolutional neural network to obtain fused features. The convolutional neural network can be a long short-term memory network. Feature extraction can reduce the dimensionality of feature data and extract information more suitable for action decision-making.

[0078] Step 203: obtaining a character action for the first virtual character based on the fusion feature.

[0079] Optionally, when determining a character action based on the fused features, the process is implemented by selecting the best action suitable for execution in the current scenario from the candidate actions. Schematically, multiple candidate actions are pre-given, and at least one candidate action is selected from the multiple candidate actions based on the fused features as the character action. For example, candidate action features corresponding to the multiple candidate actions are extracted, the feature similarity between the fused features and the multiple candidate action features is compared (e.g., determining the cosine value between the features), and at least one candidate action with the greatest feature similarity is selected as the character action.

[0080] In some embodiments, if the action decision process is performed through an action decision model, during the training process of the action decision model, the model parameters of the decision network used for action decision in the action decision model can be continuously adjusted according to the fusion features, so as to learn the best decisions in various scenarios, that is, learn the best actions in various scenarios.

[0081] Optionally, the decision network is a network layer within the action decision model, responsible for determining character actions based on a policy. The policy defines the agent's behavior for a given state, specifically, the behavior of the first virtual character for a given character state and environmental perception information. Training the action decision model can be understood as adjusting the policy within the decision network based on the rewards derived from the predicted character actions.

[0082] Schematically, in the process of making action decisions through the decision network, the probability distribution between different sub-actions in each action type is first determined based on the fusion features, and then the object sub-actions in each action type are determined based on the probability distribution.

[0083] In one possible implementation, in a game scenario, developers will set a cooling time for some sub-actions. If the cooling time has not ended, the cooling state of the sub-action will also be input into the action decision model as the character state information of the first virtual character, thereby reducing the execution probability of the sub-action in the probability distribution determined by the decision network, and avoiding the situation where the object sub-action determined by the decision model cannot be executed.

[0084] Step 204: Control the first virtual character to perform character actions.

[0085] Illustratively, the character actions determined based on the fusion features are actions controlled by the first virtual character in the virtual environment. Optionally, the character actions include at least one of a variety of actions, such as walking, running, jumping, crawling, kneeling, bending, hurdling, and shaking the head. In other words, the character actions include overall movements of various body parts of the first virtual character, as well as partial movements of at least one body part. The form of the character actions is not limited herein.

[0086] In some embodiments, taking the example of executing an action decision process and determining a character action through an action decision model, the action decision model is deployed in a second server. When the character action is determined through the action decision model, the action instruction is sent to a first server that provides background services for applications supporting virtual environments, and then the first server sends the action execution instruction to the client to achieve the purpose of controlling the first virtual character to perform the character action.

[0087] In one possible implementation, a computer device encodes the names of each sub-action in different action types respectively to obtain multiple candidate action labels. The computer device obtains the probability distribution of each sub-action under each action type based on the decision network in the action decision model, and determines the corresponding label of the object sub-action from the sub-action label based on the action execution probability. When the object sub-action label is determined, the computer device decodes the object sub-action label to obtain the object sub-action, and controls the first virtual character to perform a character action composed of multiple object sub-actions.

[0088] In summary, when a first virtual character needs to make an action decision in a virtual environment, the character state information and multimodal environmental perception information of the first virtual character are obtained, and feature encoding and fusion of the character state information and multimodal environmental perception information are then performed, ultimately making an action decision based on the fused features. Acquiring environmental perception information of different information dimensions perceived by the first virtual character when making an action decision helps to obtain a more comprehensive and detailed three-dimensional virtual environment surrounding the first virtual character, thereby enabling more reasonable action decisions. Using different encoding methods to feature encode the character state information and multiple different modal perception information enables the action decision model to more accurately fit various types of information in the current scene. Furthermore, compared to the related art method of using image recognition to obtain environmental features, extracting features from quantified multimodal environmental perception information and then fusing the encoding results of the character state information helps to more fully consider the character state information of the first virtual character while quantifying the environmental perception information. This improves the accuracy of the character action decisions while saving computing resources, making the first virtual character's execution of the role action more realistic.

[0089] In the embodiment of the present application, environmental perception information includes first modality perception information, second modality perception information, and third modality perception information. The acquisition methods for perception information of different modalities are also different. The following describes the acquisition methods for perception information of different modalities.

[0090] 1. Obtain first modal perception information.

[0091] The first modal perception information is one-dimensional vector information, and is used to represent the depth of obstacles at the same level around the first virtual character. In one possible implementation, the first modal perception information is obtained through ray detection based on the position of the first virtual character in the virtual environment.

[0092] Illustratively, the character position is used to represent the position information of the first virtual character in the virtual environment. Optionally, the character position is represented by the position coordinates of the first virtual character in the world coordinate system corresponding to the virtual environment; or the character position is represented by the position coordinates of the first virtual character in the local coordinate system corresponding to other virtual characters, etc.

[0093] Optionally, environment perception rays are emitted in multiple directions, starting from the first virtual character's position. These rays are used to detect and measure objects and scenes in the virtual environment. They simulate the working principles of devices such as light, sensors, and radar, thereby enabling functions such as object detection, collision detection, and line-of-sight detection in the virtual environment. The emission of these rays is determined by both the starting point and the emission direction. For example, the starting point is the character's position, and the emission directions are various directions in the virtual environment.

[0094] Optionally, when the environmental perception ray strikes an obstacle surface, it may be reflected in the direction in which the environmental perception ray was detected. The computer device may then obtain the reflection of the environmental perception ray and generate first modal perception information based on the reflection of the environmental perception ray. The different environmental perception rays are located at the same horizontal height.

[0095] Furthermore, to more accurately obtain first modal perception information of the virtual environment surrounding the first virtual character, the computer device emits environmental perception rays in multiple directions, starting from at least two heights at the first virtual character's location. First modal perception information is generated based on the reflection of the environmental perception rays. The emission directions of different environmental perception rays emitted at the same height are located at the same horizontal height. For example, environmental perception rays are emitted in multiple directions, starting from the feet, waist, and head of the first virtual character, to obtain first modal perception information at three different heights. The environmental perception rays emitted at multiple different heights are annular rays. Annular rays are rays that are emitted from the first virtual character itself and extend outward around itself. The multiple annular rays obtained by combining multiple different heights can be collectively referred to as multi-layer annular rays (e.g., annular rays emitted at three heights are collectively referred to as three-layer annular rays). Multi-layer annular rays can avoid the limitations of single annular rays and help to more richly and three-dimensionally represent the surrounding terrain information.

[0096] Schematically, Figure 3 shows a schematic diagram of a ray detection scheme provided by an exemplary embodiment of the present application. The computer device uses the head, waist and legs of the first virtual character as starting points, and emits environmental perception rays in multiple directions. When the environmental perception rays collide with obstacles in the virtual environment, they are reflected, thereby obtaining the reflection conditions of the environmental perception rays at three different heights and generating first modal perception information.

[0097] In one possible implementation, the number of environmental perception rays emitted from different altitudes is the same, that is, the number of annular rays in each layer is the same. The reflection of annular rays at different altitudes can represent the distance between obstacles in the virtual environment and the first virtual character at different altitudes. The greater the number of environmental rays in each layer, the more refined the perception of obstacles around the first virtual character, and the higher the performance requirements of the computer equipment.

[0098] In the embodiment of the present application, ray detection is performed using environmental perception rays at at least two heights to obtain first modal perception information. This facilitates obtaining environmental characteristics of the surrounding environment at different heights, thereby improving the comprehensiveness of environmental characteristics. Furthermore, it enables the action decision model to fully consider the height characteristics of the current virtual environment when making action decisions, helping to make the behavior exhibited by the first virtual character after executing the character action more humanized, that is, more consistent with the behavioral logic of humans in the physical environment. For example, the first virtual character can exhibit humanized behaviors such as reasonable climbing up and down stairs, thereby improving the vividness of the character animation.

[0099] 2. Obtain second modality perception information.

[0100] The second modal perception information is a two-dimensional depth map of the direction in which the first virtual character is facing. The second modal perception information is used to represent the depth of obstacles in the direction in which the first virtual character is facing.

[0101] In a possible implementation, the computer device acquires the second modality perception information through ray detection based on the character position of the first virtual character in the virtual environment and the orientation of the first virtual character.

[0102] Schematically, the orientation is used to represent the direction the first virtual character is facing in the virtual environment. For example, if the first virtual character is facing east, the orientation is represented as 0°, and if the first virtual character is facing south, the orientation is represented as 90°.

[0103] Optionally, an environment perception ray is emitted in the direction of the first virtual character, starting from the first virtual character. When the environment perception ray collides with an obstacle surface, the ray is reflected along the normal direction of the environment perception ray, so that the computer device can generate second modal perception information based on the reflection of the environment perception ray.

[0104] Illustratively, with the first virtual character as the starting point, a single environment perception ray is emitted in the direction of the first virtual character; or, with the first virtual character as the starting point, multiple environment perception rays are emitted in the direction of the first virtual character.

[0105] For illustration, please refer to Figure 4, which shows a schematic diagram of second modal perception information provided by an exemplary embodiment of the present application. The schematic diagram of the virtual environment corresponds to a two-dimensional depth map. The second modal perception information is a two-dimensional depth map in the direction of the current virtual character. The darker the pixel color in the depth map, the closer the obstacle corresponding to the pixel is to the first virtual character; the lighter the pixel color in the depth map, the farther the obstacle corresponding to the pixel is from the first virtual character.

[0106] The degree of detail of the virtual environment represented by the two-dimensional depth map is positively correlated with the resolution. The greater the sampling resolution of the two-dimensional depth map, the finer the depth information of the obstacle in front of the first virtual character represented by the two-dimensional depth map.

[0107] In an embodiment of the present application, when the environmental perception rays have reflective characteristics, a depth map in the direction of the first virtual character is obtained as the second modal perception information by emitting environmental perception rays in the direction, which is beneficial to enhancing the environmental perception ability of the first virtual character in the direction, thereby helping to significantly improve the advantages in scene understanding, object detection, visual effects, etc. with the help of the second modal perception information, achieve more accurate distance perception, environmental mapping and other environmental perception effects, and fully improve the accuracy of environmental features within the front field of view of the first virtual character, so as to provide more three-dimensional and direction-specific environmental features.

[0108] 3. Obtain third modality perception information.

[0109] The third modality of perception information is a three-dimensional occupancy grid map, which is used to characterize the spatial distribution of obstacles within a preset range. This occupancy grid map is a distribution grid used to describe the probability of obstacle distribution across the entire map. The virtual environment space is divided into multiple grids using a pre-given occupancy grid. The occupancy grid is composed of multiple cubes, each corresponding one-to-one to multiple grids. A grid can be understood as a divided portion of the virtual environment. The proportion of each grid occupied by an obstacle is detected, and all grids form a network, which is essentially an occupancy grid map. Each pixel in the map represents the probability distribution of obstacles in the real environment.

[0110] In a large-scale three-dimensional virtual environment, the occupancy grid provides macroscopic perception of the virtual environment. Optionally, the computer device, based on the first virtual character's position in the virtual environment, uses ray detection to obtain third-modal perception information of the virtual environment within a preset range of the first virtual character. The preset range is a pre-determined range based on the first virtual character's position. Optionally, the preset range is a circular area 1 meter around the first virtual character; alternatively, the preset range is a rectangular area 5 meters around the first virtual character, etc.

[0111] Optionally, first, a virtual environment within a preset range of the first virtual character is divided into an occupation grid. The occupation grid is composed of a plurality of cubes of equal volume. Then, environmental perception rays are emitted from starting points in at least two directions in the virtual environment into the occupation grid. Based on the reflection of the environmental perception rays, the computer device determines the occupation ratio of obstacles in each grid cell, thereby generating an occupation grid map.

[0112] The occupation grid can further refine the macroscopic virtual environment features surrounding the first virtual character. For a virtual environment of the same size, the more grids are divided, the more detailed the virtual environment information represented by the occupation grid. Furthermore, the more environmental sensing signals are transmitted to each grid, the more detailed the virtual environment information represented by the occupation grid.

[0113] Schematically, please refer to FIG5 , which shows a schematic diagram of a grid structure provided by an exemplary embodiment of the present application, wherein the volumes of cubes corresponding to different grids are consistent.

[0114] In an embodiment of the present application, the virtual environment is divided by a preset occupancy grid, and the multiple grids obtained by the division are composed into an occupancy grid map of the third modal perception information, which fully utilizes the simple and intuitive modeling advantages of the occupancy grid map to better characterize the macro-environmental characteristics around the first virtual character. The data results of the occupancy grid map are usually implemented in the form of a three-dimensional array, so the effect of accessing and analyzing the occupancy grid map is also higher, and not only the third modal perception information can be obtained in advance, but also the third modal perception information can be obtained in real time. In addition, by controlling the degree of detail of the occupancy grid, the macro-environmental characteristics can be further refined, thereby characterizing more detailed obstacle distribution.

[0115] The above introduces the methods of obtaining first modal environmental information, second modal environmental information and third modal environmental information. Although the acquisition of environmental information of different modalities depends on the position of the first virtual character in the virtual environment, due to the dimensional differences obtained during ray detection, the virtual environment can be more comprehensively analyzed from one-dimensional, two-dimensional and three-dimensional situations, thereby improving the comprehensiveness of the acquisition of environmental perception information, helping to improve the character perception ability in the virtual environment, and enhancing the interactive effect of the virtual environment.

[0116] In an optional embodiment, a method for executing a virtual character's action decision is implemented using an action decision model. The action decision model includes an encoding network and a decision network. The encoding network is used to encode and fuse features of character state information and environmental perception information, while the decision network is used to determine the character's action based on the fused features. Furthermore, the encoding network includes a perception information encoder and a state information encoder. The decision network includes n action output heads, each of which is used to serially output n object sub-actions. Different action output heads correspond to different action types, and the n action output heads are serially connected based on a dependency relationship between the action types. The dependency relationship is used to characterize the dependency restrictions between sub-actions under different action types. The n object sub-actions constitute the character action.

[0117] In the action decision model, n action output heads belong to the output layer of the action decision model, which are used to predict actions in different dimensions, thereby outputting action parameters of different dimensions. Among them, the action dimensions supported by the first virtual character may include movement state dimension, movement direction dimension, orientation dimension, attack dimension, and posture dimension, etc. In the action decision model, n action output heads correspond to n action dimensions respectively, and then the n action output heads output action parameters of n different dimensions. For example, the action parameter output by the action output head corresponding to the movement direction dimension is a movement direction, such as a 90° direction; the action parameter output by the action output head corresponding to the orientation dimension is a turning angle that can represent the direction, such as -5°, representing a 5° rotation in the counterclockwise direction; the action parameter output by the action output head corresponding to the attack dimension is attack or not attack, etc. This embodiment does not limit the division of action dimensions and the action parameters output by the action output heads corresponding to different action dimensions.

[0118] The process of generating character actions through an action decision model will be described below through an exemplary embodiment.

[0119] Please refer to FIG6 , which shows a flowchart of a process for generating character actions provided by an exemplary embodiment of the present application.

[0120] Step 601: Input the character state information and the environment perception information into the action decision model, encode the character state information and the environment perception information through the encoding network, and obtain the state information encoding result corresponding to the character state information and the perception information encoding result corresponding to the environment perception information.

[0121] Optionally, the state information encoder encodes the character state information to obtain a state information encoding result; and the perception information encoder encodes the environment perception information to obtain a perception information encoding result. The two encoding processes can be performed sequentially (e.g., encoding by the state information encoder first, then encoding by the perception information encoder; or encoding by the perception information encoder first, then encoding by the state information encoder) or simultaneously, without limitation herein.

[0122] Optionally, the perception information encoder includes convolution kernels of different dimensions, and the convolution kernels of different dimensions are used to perform feature encoding on perception information of different modalities. Optionally, the first modal perception information is encoded by a one-dimensional convolution kernel in the perception information encoder to obtain a first perception information encoding result. In order to enable the action decision model to perceive the three-dimensional virtual environment information around the first virtual character, a circular convolution method is used to obtain environmental perception information of the virtual environment around the first virtual character based on the distance between the obstacle represented by each environmental perception ray and the first virtual character.

[0123] Schematically, Figure 7 shows a schematic diagram of the one-dimensional convolution process for environmental perception information, provided by an exemplary embodiment of the present application. This process includes three different heights of environmental perception rays, with each layer of annular rays comprising 144 environmental perception rays. Each layer of environmental perception rays has one leading ray and one trailing ray. Because annular rays are connected head to tail, a one-dimensional convolution kernel is used to perform circular convolution calculations on the corresponding environmental perception rays in the three layers of annular rays, thereby encoding the first modal perception information.

[0124] In addition, during the training of the motion model, feature encoding of three environmental perception rays at different heights is performed through one-dimensional circular convolution, which can avoid missing valuable information, accelerate the training process, and improve the training effect.

[0125] Optionally, the second modal perceptual information is encoded using a two-dimensional convolution kernel in a perceptual information encoder to obtain a second perceptual information encoding result. The two-dimensional convolution kernel is a two-dimensional matrix. During the process of encoding the second modal perceptual information using the two-dimensional convolution kernel, feature encoding of the second modal perceptual information is completed by performing element-by-element multiplication and summation of the two-dimensional matrix and the input two-dimensional depth map.

[0126] The process of element-by-element multiplication and summation is schematically explained. Assume there is a two-dimensional matrix A and a two-dimensional depth map of the same size, and the following operations are performed on the elements of both at the same position:

[0127] (1) Element-wise Multiplication:

[0128] In element-by-element multiplication, the elements at corresponding element positions are multiplied to obtain a new matrix C, in which each element in the matrix C is the product of the element at the corresponding position of the matrix A and the depth map.

[0129] (2) Summation:

[0130] Add all elements in the matrix C to get a scalar result S, which represents the sum of all products, that is, the result of the element-by-element multiplication and summation.

[0131] Optionally, the third modal perception information is encoded by a three-dimensional convolution kernel in the perception information encoder to obtain a third perception information encoding result, and the first perception information encoding result, the second perception information encoding result and the third perception information encoding result constitute the perception information encoding result. By extracting features from the occupancy grid map through a three-dimensional convolution kernel, macroscopic three-dimensional scene perception information can be provided to the decision network in the action decision model. For example, the first virtual character has a preset task in the game scene. To complete the preset task, the first virtual character needs to move in a large-scale map. In this case, in order to train the action decision model used in the scene, encoding the third modal perception information through a three-dimensional convolution kernel is beneficial to providing the action decision model with macroscopic scene perception information.

[0132] The above content describes the process of encoding different modal perceptual information through convolution kernels of different dimensions and obtaining the corresponding perceptual information encoding results. Setting convolution kernels of different dimensions facilitates the differentiated analysis of perceptual information of different modalities, and also facilitates the targeted and unique analysis of perceptual information of modalities under the same dimension, thereby better preserving and utilizing the characteristics of each modality. Combining perceptual information of multiple modalities encoded by different convolution kernels can effectively integrate perceptual information, so that environmental perceptual information expresses stronger overall perception and environmental understanding capabilities, avoiding the unreliability caused by harsh conditions such as noise and occlusion in a single modality, and helping to obtain relatively accurate and adaptable environmental perceptual information in complex virtual environments.

[0133] Step 602: The splicing state information encoding result and the perception information encoding result are concatenated to obtain a splicing feature code.

[0134] After obtaining the state information coding result, the first environment perception information coding result, the second environment perception information coding result and the third environment perception information coding result, the state information coding result and the perception information coding result are feature spliced ​​through a fully connected layer and an activation function to obtain a spliced ​​feature code.

[0135] Schematically, according to the splicing order of the state information encoding result, the first environmental perception information encoding result, the second environmental perception information encoding result and the third environmental perception information encoding result, a feature splicing process is performed to obtain a spliced ​​feature code; or, according to the splicing order of the first environmental perception information encoding result, the second environmental perception information encoding result and the third environmental perception information encoding result and the state information encoding result, a feature splicing process is performed, etc. The splicing order is not limited here. The fully connected layer is used to map the feature space obtained by the previous convolution calculation to the sample label space (that is, to integrate the feature representation into one value), which can reduce the influence of the feature position on the classification result and improve the robustness of the action decision model. The activation function is used to add nonlinear factors to improve the expressive power of the action decision model.

[0136] Step 603: extract and concatenate feature codes through coding network features to obtain fusion features.

[0137] In one possible implementation, the encoding network also includes an LSTM (Long Short Term Memory) network. Feature extraction is performed by inputting the fused features into the LSTM network to obtain the fused features. During the feature extraction process, the LSTM network retains the more valuable features and forgets the less valuable features, thereby achieving feature extraction.

[0138] Optionally, the LSTM network can be replaced by other convolutional neural networks, which is not limited in this embodiment.

[0139] Steps 601 through 603 above describe the acquisition of fused features through the encoding network in the action decision model. Encoding character state information and environmental perception information, respectively, through the state information encoder and perception information encoder in the encoding network helps leverage the targeted analysis characteristics of different encoders. This improves the accuracy of character state information and refines the granularity of character state analysis, while more specifically capturing various perceptual information in the virtual environment. This allows for better resource management and task allocation, and more efficient acquisition of more accurate fused features.

[0140] In step 604 , the fused features are input into the first action output head in the decision network, and the first object sub-action is determined from the sub-actions of the first action type through the first action output head in the decision network.

[0141] Inputting the fused features into the decision network, that is, inputting the fused features into the first action output head of the decision network, and determining the first object sub-action from the sub-actions in the first action type through the first action output head in the decision network.

[0142] Optionally, the executable actions of the first virtual character are orthogonally decomposed to obtain n action types, each of which contains at least two executable actions. The sub-actions under different action types can be independently controlled, that is, the first virtual character can be controlled to only execute sub-actions in one action type. The orthogonal decomposition is used to divide the executable actions of the first virtual character into multiple discrete action types. The n action types obtained after the orthogonal decomposition are mutually orthogonal action types. There is no intersection between the sub-actions in different orthogonal action types, that is, there is no situation where a sub-action belongs to two action types. Different orthogonal action types support simultaneous execution, and different sub-actions in the same orthogonal action type do not support simultaneous execution.

[0143] In the game scene, there are the most basic and fine-grained atomic actions to control the first virtual character, and such actions cannot be further split and refined. The sub-actions in an action type are atomic actions, and different sub-actions in the same action type cannot be executed at the same time. For example, if the first virtual character can perform actions including crawling forward, this action can be divided into two atomic actions, crawling and moving forward, where crawling corresponds to the posture action type, and moving forward corresponds to the moving direction action type. For another example, if the first virtual character can perform actions including standing still and peeking and shooting, this action can be divided into three atomic actions, staying in place, peeking, and shooting, where staying in place corresponds to the moving state action type, peeking corresponds to the peeking action type, and shooting corresponds to the attacking action type.

[0144] For illustration, please refer to Figure 8, which shows a user interface diagram of an application providing a virtual environment, according to an exemplary embodiment of the present application. The user interface includes a movement state control 801, an attack control 802, a posture adjustment control 803, a horizontal steering control, a vertical steering control 804, a left and right probe control 805, and a movement state control 806. Different controls can be used to control the virtual character to perform different actions.

[0145] Performing an orthogonal decomposition of the actions corresponding to the various controls shown in the above figure yields a variety of action types. In some embodiments, dependencies may exist between different action types. For example, standing or leaning sideways is performed only when an attack is required, so the character's posture action decision needs to rely on the action decision for the offensive action type. Furthermore, dependency restrictions indicate that there may be dependency restrictions between sub-actions within different action types. For example, when the first virtual character performs a sprint action, simultaneous offensive actions are not supported, so a dependency restriction exists between the sprint action and the offensive action. Therefore, to enable the action decision model to consider the dependencies between different actions, as well as the dependency restrictions between different sub-actions within different action types, when making action decisions, the n action output heads in the decision network are serially connected based on the dependencies between action types.

[0146] Optionally, when determining the order in which different action output heads are connected, the action types are sorted based on their relevance to the task to be completed by the first avatar. For example, if the first avatar's task is to defeat an enemy, the attacking action type may be determined to have a higher relevance, and its corresponding action output head may be arranged at the front. However, the sideways action type may have a lower relevance, and its corresponding action output head may be arranged at the back.

[0147] Illustratively, Table 2 shows the correspondence between action output heads and action types after orthogonal decomposition provided by an exemplary embodiment of the present application.

[0148] Table 2

[0149] In some embodiments, there is a need for the first virtual character to aim at other virtual characters, and other virtual characters may serve as opponents of the first virtual character. That is, when the first virtual character has the need to aim at opponents, if a steering design scheme that is evenly distributed within the steering range is adopted, it will be difficult for the first virtual character to quickly and accurately aim at the enemy when the granularity of the angle is too large or too small. Therefore, precise aiming can be achieved by combining coarse-grained steering angles and fine-grained steering angles. The corresponding angle interval of the sub-actions of the coarse-grained steering angle is the first interval, and the corresponding angle interval of the sub-actions of the fine-grained steering angle is the second interval. The product of the first interval and the coarse-grained horizontal direction action dimension is the current field of view of the first virtual character, and the current field of view changes according to the moving direction of the first virtual character. The product of the second interval and the action dimension is equal to the first action interval.

[0150] Please refer to Figure 9, which shows a schematic diagram of the movement direction and sub-actions describing the action provided by an exemplary embodiment of the present application, wherein the movement direction action types are divided into eight different angles, distributed within a 360° range based on the center area of ​​the first virtual character's current orientation. The turning angle includes a horizontal turning angle and a vertical turning angle, wherein the horizontal turning angle includes a coarse-grained horizontal turning angle and a fine-grained horizontal turning angle. The coarse-grained horizontal turning angle is based on the current orientation of the first virtual character, the current orientation is set to 0°, the turning range is 120 degrees, and the interval between each sub-action of the coarse-grained turning angle is 20°, that is, the turning angles corresponding to the sub-actions in the coarse-grained turning angle are 0°, 20°, 40°, 60°, -20°, -40°, and -60° respectively. If it is determined that the object sub-action of the coarse-grained turning angle corresponds to 40°, after executing the turning action, the coarse-grained horizontal orientation of the first virtual character is 40°. The fine-grained steering angle is based on the estimated orientation of the first virtual character after pre-execution of the coarse-grained sub-action, which is 0°. It is evenly distributed within a set range, which can be 20. The interval between each sub-action of the fine-grained steering angle can be 3°. The steering angles corresponding to the sub-actions included in the fine-grained steering angle are 3°, 6°, 9°, ..., 21°, respectively. The sub-action corresponding to the object of the fine-grained steering angle is determined to be a 6° turn to the right. For vertical orientations with a smaller range, the fine-grained steering angle can be used to divide the sub-actions. The current orientation of the virtual character is used as the reference, which is 0°. The setting range is -30°-45°, and the interval between sub-actions can be set to 3°.

[0151] Optionally, the clockwise rotation angle is a positive value, the counterclockwise rotation angle is a negative value, the upward rotation angle is a positive value, the downward rotation angle is a negative value, or vice versa, which is not limited in this embodiment.

[0152] In step 605, the embedded coding vectors corresponding to the determined first object sub-action to the i-1th object sub-action and the fusion features are input into the i-th action output head, and the i-th object sub-action is determined from the sub-actions of the i-th action type through the i-th action output head.

[0153] Wherein, n is a positive integer greater than 2, and i is a positive integer greater than 1.

[0154] The n action output heads are connected through an autoregressive embedding layer. Since there is a dependency relationship between different action types, after the preceding output head determines the object sub-action, the at least one determined object sub-action and the fusion vector should be input into the subsequent action output head based on the autoregressive embedding layer, so that the subsequent action output head determines the object sub-action according to the dependency relationship.

[0155] Autoregression is a statistical method for processing time series, using the performance of the same variable at previous moments to predict the current moment. Autoregression exhibits autocorrelation in time series. The essence of embedding is data compression, using lower-dimensional features to represent higher-dimensional features with redundant information.

[0156] Please refer to Figure 10, which shows a schematic diagram of the autoregressive structure of n action output heads provided by an exemplary embodiment of the present application. An action output head includes a fully connected layer, a Logits layer, a sampling layer, and an embedding layer. The Logits layer is the execution probability distribution between each sub-action in the determined action type, thereby determining the object sub-action. After the object sub-action is determined, the determined sub-action is transmitted to the next serially connected action output head through the embedding layer, and the fusion layer in the action output head fuses the various object sub-actions that have been determined in the previous order, thereby determining a new object sub-action based on the determined object sub-action. In addition, the encoding network will also input the fused features into each action output head.

[0157] In an embodiment of the present application, environmental perception information of different modalities (i.e., different information dimensions) is encoded based on convolution kernels of different dimensions, so that the environmental information of each information dimension can be processed in a suitable encoding method, thereby improving the feature expression of the encoded environmental perception information, thereby enabling the action decision model to obtain more accurate environmental features.

[0158] In the embodiment of the present application, the content of action decision making through the decision network in the action decision model is also introduced. With the help of the action output head in the decision network, the first object sub-action can be determined from the sub-actions of the action type. By integrating the embedded coding vectors of the first i-1 object sub-actions and fusing the features to the i-th action output head, it helps to more comprehensively determine the i-th object sub-action. This structure helps the system make efficient decisions and action plans in complex environments. It uses the fused multimodal information to guide the selection of each action step, and at the same time transmits the results of the previous decision through the embedded coding vector to ensure the coherence and rationality of subsequent actions.

[0159] Since sub-actions under different action types have dependency restriction relationships, and at least two sub-actions with dependency restriction relationships do not support simultaneous execution, then in the case of a determined preceding object sub-action, some sub-actions in the subsequent action type may have dependency restriction relationships with the determined object sub-action. Moreover, since the dimension of the action space after orthogonal decomposition is still very high, it will lead to a rich explosion of anthropomorphic atomic action space combinations. Therefore, sub-actions can be determined based on the rationality of the combination between different sub-actions (i.e., dependency restriction relationships), thereby improving the anthropomorphism of the action combination. In addition, in the process of training the action decision model, action masking can also be used to reduce the decision network from exploring invalid decision strategies, thereby accelerating training efficiency and improving the anthropomorphism of the action decision model in making action decisions.

[0160] Action masking can be used in reinforcement learning models (corresponding to the action decision model in the embodiment of the present application), and can screen out unreasonable or invalid sub-action sets in large-scale decision spaces, thereby reducing meaningless action exploration and enabling the decision network to converge faster. In essence, action masking is used to screen out sub-actions in different action types that are unreasonable or have a dependent and restrictive relationship with the determined object sub-actions. By means of action masking, the probability of the sub-actions that need to be masked in the probability distribution obtained by the action output head is reduced, so that when the object sub-action is determined by sampling according to the probability, it is difficult to determine the masked sub-action as the object sub-action. By means of action masking, unnecessary exploration sub-action branches can be directly excluded during the action decision process, thereby reducing the action space and ensuring the strength of the action decision model while improving the exploration efficiency of the action decision model.

[0161] Among them, at least two sub-actions with a dependency restriction relationship belong to different action types, and at least two sub-actions with a dependency restriction relationship cannot be executed at the same time. Optionally, if there is a dependency restriction relationship between the first sub-action and the second sub-action, the two sub-actions cannot be executed at the same time, and when it has been determined that the first sub-action is a character action, the second sub-action is action-masked. Optionally, if there is a dependency restriction relationship between the second sub-action and the third sub-action after it is combined with the fourth sub-action, the combination of the second sub-action and the third sub-action and the fourth sub-action cannot be executed at the same time, and when it has been determined that the second sub-action and the third sub-action are both object sub-actions, the fourth sub-action is action-masked.

[0162] Optionally, the action decision model includes an action mask network, which is connected to n action output heads in the decision network and is used to perform action masking on sub-actions in each action type during the action decision process of each output head.

[0163] In some embodiments, before determining the character action, the computer device masks the sub-actions in the i-th action type corresponding to the i-th action output head based on the determined first to i-1th object sub-actions, as well as the dependency constraints between different sub-actions in different action types. The computer device then determines the i-th object sub-action from the unmasked sub-actions in the i-th action type using the i-th action output head.

[0164] Optionally, when performing action masking, in the network's original logits action output layer, by adding a large negative number to the logits of unreasonable sub-actions, the logits value corresponding to the action is guaranteed to be smaller than the logits values ​​of all reasonable sub-actions, thereby reshaping the logits distribution of the output action layer and mapping the probability of unreasonable actions being sampled close to zero.

[0165] The above content describes how the decision network optimizes sub-action selection through action masks and dependency constraints. By applying action masks, we can eliminate unreasonable or invalid sub-action selections based on the dependencies and constraints between sub-actions within different action types. This helps ensure that the selected sub-actions are logically coherent and valid, and avoids conflicting and inconsistent action selections, thereby ensuring the rationality of the overall action. Effective action selection also means that appropriate actions can be executed more quickly and efficiently, saving resources and optimizing the overall execution process.

[0166] Before performing action masking, first, based on the determined object sub-actions, among the action types to be decided, determine the sub-actions to be masked that have a dependency restriction relationship with the determined object sub-actions. Specifically, this can include the following three situations:

[0167] 1. There is a determined object sub-action that has a dependency restriction relationship with the sub-action to be masked in the action type to be decided.

[0168] In one possible embodiment, when there is a determined j-th object sub-action and the dependency restriction situation indicates that the j-th object sub-action has a dependency restriction relationship with at least one first sub-action in the i-th action type, the first sub-action in the i-th action type is action masked, and j is less than i (that is, the j-th action head is connected in series before the i-th action head).

[0169] For example, there is a determined sprint action, and the dependency restriction situation indicates that there is a dependency restriction relationship between the sprint action and the squat action. Then, when the action type of the first virtual character posture is determined, the squat action in the posture action type is action masked, and j is a positive integer greater than 2.

[0170] Optionally, the computer device may perform one-hot processing on the object sub-action determined by the preceding action output header, and then perform matrix multiplication on the one-hot matrix corresponding to the preceding object sub-action based on a restriction matrix representing the dependency restriction relationship between each sub-action in the subsequent action type and the sub-action in the preceding action type, thereby obtaining an action mask matrix corresponding to the subsequent action type. Illustratively, the computer device transmits the action determined by the preceding action output header to the subsequent action output header based on the action mask matrix.

[0171] Schematically, there are action type a1 for initiating an attack (including attacking and not attacking), action type a2 for the avatar's posture (including squatting, jumping, crawling, and standing), and action type a7 for the avatar's movement (including standing still, walking fast, sprinting, and standing still). Action type a1 corresponds to action output head H1, action type a2 corresponds to action output head H2, and action type a7 corresponds to action output head H7. Attacking and sprinting have a dependency constraint, and squatting also has a dependency constraint on sprinting. Because the values ​​of sub-actions flow in the neural network as tensors, it is not possible to directly sample the object sub-actions to avoid gradient disconnection.

[0172] Therefore, the sample is first determined as a one-hot matrix τ n×m Assume that there are n = 3 samples, the action dimension of a1 (i.e., the number of sub-actions it contains) is m = 2, and the output sub-actions of the action output head H1 of the three samples are {attack, do not attack, attack}, then the n × m one-hot matrix τ corresponding to these three samples is n×m for:

[0173] τ n×m It is the one-hot matrix output by the action output head H1 in the sample. The number of rows is the number of samples n, and the number of columns represents the number of sub-actions m contained in the action type a2.

[0174] Next, we construct an auxiliary mapping matrix M, which represents the dependency constraint relationship between the sub-actions in action type a1 and the sub-actions in action type a7. Since the mapping relationship for action type a7 is constructed based solely on the object sub-action samples output by a single action output head H1, the one-hot matrix composed of all the different actions in action output head H1 begins as an m×m identity matrix. The first row maps the attacking object sub-actions, and the second row maps the non-attacking object sub-actions.

[0175] Since the action output head H7 corresponds to dimension P = 4, in order to reflect the attack without sprinting, the auxiliary mapping matrix M can be obtained m×p for:

[0176] M m×p The essence of the matrix is ​​a mapping from H1 to H7. The number of rows represents the dimension m of the preceding action head H1, and the number of columns represents the dimension P of the subsequent affected action head H7. Each row is related to the unit matrix E. m×m Each row in corresponds to an association, and the specific value of each row refers to the impact of each sub-action in the previous action output head H1 on the subsequent action output head H7.

[0177] Finally, perform matrix multiplication to convert τ in the sample n×m The effect of not being able to sprint when launching an attack is mapped to the action output head H7, and the action mask matrix X7 corresponding to the action output head H7 is obtained as follows:

[0178] The action mask matrix X7 is used to characterize the influence of the object sub-action output by the preceding action output head H1 on the sub-action in the action type corresponding to the subsequent action output head H7 in all samples. It can be seen that the size of X7 is only related to the number of samples and the dimension of the action output head H7. The number of rows represents the sampling of 3 samples, and the number of columns represents the different sub-actions in the action type a7. The above process has masked the sprint sub-action corresponding to the attack of the object sub-action determined in all samples, and then the probability distribution of the sub-action in the action type a7 determined by the action output head H7 is converted into a probability distribution, which can ensure that the action output head H7 will not sample the sprint action.

[0179] Similarly, action type a2 also uses the action mask matrix Y7 of the object sub-action in action type a2 on action type a3 obtained by the above processing. Since action types a1 and a2 both independently affect action type a7, the action output head H7 superimposes the influence of the object sub-action output by the previous action on the sub-action in action type a7, so the action mask matrix L7 corresponding to the action output head H7 is: L7 = X7○Y7

[0180] The L7 matrix is ​​the Hadamard product of X7 and Y7. All elements of L7 with a value of 0 map to the action type a7, which will be masked.

[0181] Based on the above method, the influence of the object sub-action determined in the previous order on the sub-action in the action type corresponding to the subsequent action output head can be mapped to the action mask matrix, so that the influence is transmitted backward in the n action output heads in sequence. This can effectively solve the problem of a large amount of invalid exploration of the decision model in the high-dimensional action space during the training process of the action decision model.

[0182] 2. There are at least two determined object sub-actions that, after being combined, have a dependency restriction relationship with the sub-action to be masked in the action type to be decided.

[0183] In another possible embodiment, there may be determined object sub-actions, and these two object sub-actions do not have an independent impact on the sub-actions in the subsequent action type. After at least two determined object sub-actions are combined, there will be a dependent restriction relationship with the sub-actions in the action type to be decided. For example, there are two action types: moving direction and horizontal turning angle. The sub-actions in these two action types will not affect the sprint action in the action type to be determined alone. However, in the case where the difference between the horizontal turning angle and the moving direction in the determined object sub-action is large (180° in extreme cases), for example, the first virtual character's gaze direction is 0° but the moving direction is 180°. Since the first virtual character is not supported to sprint backwards in the conventional setting, the first virtual character does not support the simultaneous execution of the sprint action.

[0184] In this case, the object sub-actions output by at least two action output heads should be combined to determine the impact of the combination of at least two object sub-actions on the sub-actions in the action type to be decided, and the impact should be mapped into the action mask matrix corresponding to the action type to be decided.

[0185] Optionally, each of the object sub-actions output by the at least two action output heads is first encoded with a one-hot matrix, and then the at least two one-hot matrices are concatenated to form a merged one-hot matrix. The merged one-hot matrix is ​​then logically transformed to map it to a unit matrix. Subsequently, based on the method shown in the above-mentioned method 1, an auxiliary mapping matrix is ​​determined, and based on the auxiliary mapping matrix, the influence of the combination of at least two object sub-actions on the action type to be decided is achieved after mapping each row (each action), thereby realizing the action masking of the sub-actions in the action type to be decided.

[0186] Schematically, there is action type a3 for movement direction (including eight movement directions evenly distributed within 360°), action type a4 for horizontal orientation (including eight orientation angles evenly distributed within 360°), and action type a7 for the movement state of the virtual character (including still walking, fast walking, sprinting, and standing still). Action type a3 corresponds to action output head H3, action type a4 corresponds to action output head H4, and action type a7 corresponds to action output head H7.

[0187] Among them, the action type a3 corresponds to the action dimension s, the action type a4 corresponds to the action dimension h, and the different sampling sub-actions of the action output head H3 are combined into a unique hot matrix s×s unit matrix E s×s , the one-hot matrix composed of all different sampled actions of the action output head H4 is an h×h identity matrix E h×h . For the identity matrix E s×s and the identity matrix E h×h The merged one-hot matrix T obtained by feature concatenation(s×h)×(s+h) It is not a square matrix. The number of rows is s×h, which represents all action combinations. The number of columns is s+h, which represents the combined dimension, i.e., the sum of the number of sub-actions of action type a3 and action type a4. (s×h)×(s+h) Converted to an identity matrix E of dimension s×h (s×h)×(s×h) , then each row of the identity matrix represents a combination of actions, and then the mapping auxiliary matrix M can be determined according to the dependency restriction relationship.

[0188] Optionally, assume that the merged one-hot matrix corresponding to the sub-actions output by the action output head H3 and the action output head H4 in all samples is τ n×(s+h) , the action mask matrix corresponding to the action output head H7 can be determined by the following calculation:

[0189] First, calculate the mapping of each action combination in the sample into an approximate one-hot matrix: U n×(s×h) =τ n×(s+h) ×X (s+h)×(s×h)

[0190] Among them, X (s+h)×(s×h) T (s×h)×(s+h) The generalized inverse matrix, U n×(s×h) A mapping that transforms each action combination in the sample into an approximate one-hot matrix, since X (s+h)×(s×h) Not T (s×h)×(s+h) The exact inverse matrix requires U n×(s×h) Convert to an exact one-hot matrix: Z n×(s×h) =one-hot(argmax(U n×(a×h) ))

[0191] Among them, Z n×(s×h) Each row represents an action combination of a sample. Each row contains s×h matrix values, with only one element value being 1 and the rest being 0. The element value of 1 indicates that the action combination will be associated with the corresponding row of the mapping auxiliary matrix M, thereby affecting the matrix value of the corresponding row of the action mask matrix corresponding to the action output head H7. n×p is the action mask matrix in the action output head H7, and each row represents the impact of action type a3 and action type a4 on the sub-actions in action type a7.

[0192] Through the above calculations, different sub-actions can be combined to achieve action masking when there is a dependency restriction relationship between sub-actions in the decision action type, thereby solving the problem of low exploration efficiency of the decision network in a strongly coupled high-dimensional complex action space.

[0193] 3. The character status information and environmental perception information indicate the current virtual environment situation, and have a dependency restriction relationship with the to-be-masked sub-action in the to-be-decided action type, that is, it is not suitable to execute the to-be-masked sub-action in the current virtual environment.

[0194] In one possible implementation, the first virtual character may have tasks such as attacking other virtual characters or avoiding other virtual characters. Therefore, when making action decisions, the sub-actions in the action type to be decided may be masked based on the visibility of the first virtual character to other virtual characters.

[0195] Optionally, starting from the eye position of the first virtual character, an environmental perception ray is emitted toward various parts of other virtual characters in the direction of the first virtual character. Based on the reflection of the environmental perception ray, the visibility of various parts of the other virtual characters in the virtual environment is determined. The visibility is used to indicate whether various parts of the other virtual characters are obscured by cover. If the environmental perception ray is obscured by an obstacle, the part of the other virtual character is not visible. If the environmental perception ray is not obscured by an obstacle and collides with the other virtual character, the part of the other virtual character is visible.

[0196] Please refer to Figure 11, which shows a schematic diagram of visual situation perception provided by an exemplary embodiment of the present application. Starting from the eyes of the first virtual character 1101, environmental perception rays are emitted toward various parts of the opponent virtual character 1102. If some of these perception rays are blocked by obstacles 1103, the body parts of the opponent virtual character corresponding to these rays will not be visible. However, if some perception rays 1104 are not blocked by the obstacles and collide with the opponent virtual character's head, the opponent virtual character's head will become visible.

[0197] The above describes the process of optimizing actions in a virtual environment based on environmental perception rays and visual conditions. By considering visual obstructions and visibility in the virtual environment, the virtual character's actions can be more intelligently determined, more consistent with real-world behavior logic. This improves the authenticity and realism of the virtual character's interactions in the virtual environment, providing players with a more immersive experience. The virtual character can also flexibly adjust its actions based on real-time changes in the surrounding environment and the movement or disappearance of obstacles, enhancing its autonomy in the virtual environment.

[0198] After determining the visibility of various parts of other virtual characters, the computer device performs action masks on the sub-actions in the character action type based on the visibility of various parts of other virtual characters.

[0199] The object sub-action is dependent on the visibility of various parts of other virtual characters. For example, if the character action type is a shooting action, if the visibility of various parts of other virtual characters indicates that the other virtual characters are not visible, the "shooting" sub-action in the character action is masked to prevent the first virtual character from frequently shooting without aiming at the enemy.

[0200] After masking the sub-actions in the character action type, the computer device determines the sub-actions from the unmasked sub-actions in the character action type through the character action output head.

[0201] In another possible implementation, the serial order of n action output heads is determined based on the dependencies between the corresponding action types. For the same action type, but with different fine-grained sub-actions corresponding to two action output heads, the approximate action range can be determined based on the coarse-grained action, and then the fine-grained object sub-action can be determined based on the determined coarse-grained object sub-action.

[0202] In one possible implementation, at least two of the n action output heads correspond to the same action type, the i-th action output head corresponds to a coarse-grained sub-action, and the j-th action output head corresponds to a fine-grained sub-action, where i is less than j. In determining the object sub-action, the computer device first determines the i-th object sub-action from the coarse-grained sub-actions using the i-th action output head in the decision network, and then determines the j-th object sub-action from the fine-grained sub-actions using the j-th action output head based on the fusion vector and the embedded coding vector corresponding to the i-th object sub-action, where the action range indicated by the j-th object sub-action is smaller than the action range indicated by the i-th character action.

[0203] For example, with respect to the coarse-grained horizontal orientation and the fine-grained horizontal orientation in Table 2, when making an action decision, the coarse-grained horizontal orientation should be determined first and then the fine-grained horizontal orientation.

[0204] If the first virtual character needs to aim at other virtual characters, the steering angle range that can aim at the opponent virtual character can be determined first, and then the steering angles that cannot enable the first virtual character to aim at the opponent virtual character in the steering action type can be masked.

[0205] Please refer to Figure 12, which shows a schematic diagram of determining the effective vertical aiming range provided by an exemplary embodiment of the present application. First, the distance d between the first virtual character and the opponent virtual character is determined based on the position coordinates of the opponent virtual character and the position coordinates of the first virtual character, and the height h of the opponent virtual character is obtained, h = h_z - h_f. If the crosshairs of the virtual props held by the first virtual character fall within the height range of the first virtual character in the vertical direction, the opponent virtual character can be aimed at. According to the trigonometric function, we can obtain: θ max =actan[(h_z-h_f) / d] θ min =actan[(h_f-h_z) / d] τ pitch =θ max -θ min

[0206] Among them, the horizontal direction is the current direction, which is 0°, θ max is the effective aiming range of upward rotation, θ min is the effective aiming range of downward rotation, the difference between the two is τ pitch This is the effective aiming range for vertical steering movements.

[0207] For the horizontal turning action type, we can first roughly determine the coarse-grained turning action that can target the opponent's virtual character, and then adjust the fine-grained turning action to achieve the purpose of targeting the opponent's virtual character.

[0208] Schematically, among the n action output heads, there is a first steering action output head corresponding to a coarse-grained steering action, and a second steering action output head corresponding to a fine-grained steering action. The first steering action output head is used to determine a first steering angle, and the second steering action output head is used to determine a second steering angle.

[0209] Illustratively, the first steering angle is a coarse-grained steering angle determined based on a plurality of coarse-grained steering angles; the second steering angle is a fine-grained steering angle determined based on a plurality of fine-grained steering angles.

[0210] First, the computer device determines an ideal turning angle based on the orientation of the first virtual character, the position coordinates of the first virtual character, and the position coordinates of the other virtual characters, and determines an aiming distance between the first virtual character and the other virtual characters. The computer device first calculates the aiming distance based on the position coordinates of the first virtual character and the position coordinates of the other virtual characters, and then determines the ideal turning angle based on the current orientation and position of the first virtual character based on the orientation of the other virtual characters relative to the first virtual character using trigonometric functions.

[0211] Then, based on the aiming distance and the width of other virtual characters, the effective aiming range is determined.

[0212] Schematically, please refer to Figure 13, which shows a schematic diagram of the horizontal effective aiming range provided by an exemplary embodiment of the present application. The current orientation of the first virtual character is 0°, the distance between the first virtual character and the other virtual characters is d, and the width of the opponent virtual character is W. Then, the effective aiming range is determined as: θ = actan[(W / 2) / d] τ pitch =2θ

[0213] Where θ is an effective rotation range of the first virtual character in the clockwise and counterclockwise rotation angles. Combining the clockwise rotation effective range with the counterclockwise rotation effective range, the effective aiming range τ is obtained. pitch .

[0214] After determining the ideal steering angle, the difference between the first steering angle and the ideal steering angle is determined based on the first steering angle determined by the first steering action output head. This difference may be positive or negative. A positive value indicates that after the first steering angle is applied, the first virtual character is expected to be oriented clockwise relative to the other virtual characters. A negative value indicates that after the first steering angle is applied, the first virtual character is expected to be oriented counterclockwise relative to the other virtual characters.

[0215] Optionally, the computer device determines the effective coarse-grained steering angle based on the ideal steering angle, and performs action masking on the steering angles in the coarse-grained steering action of the virtual character that do not belong to the effective coarse-grained steering angles. Then, the first steering angle is determined from the unmasked sub-actions. Subsequently, the computer device determines the effective fine-grained steering angle based on the angle difference and the effective aiming range. The determined effective fine-grained steering angle is the orientation within the effective aiming range after the first virtual character performs the steering action according to the fine-grained steering angle under the current orientation. Then, the steering angles in the coarse-grained steering action that do not belong to the effective fine-grained steering angles are action masked, and the second steering angle is determined from the unmasked effective fine-grained steering angles in the fine-grained steering action through the second steering action output head. Finally, after the first virtual character performs the first steering action and the second steering action, it can aim at other virtual characters.

[0216] The above content introduces the optimization of coarse-grained and fine-grained control during the steering process of the first virtual character in a virtual environment. Through coarse-grained and fine-grained steering control, the first virtual character's steering behavior can be made more natural and intelligent, closer to real-world behavior. Optimized steering control can enhance the first virtual character's tactical effectiveness and combat strategy in the game, and enhance the immersive gaming experience. Motion masking technology can also reduce unnecessary calculations and steering operations, improving action execution efficiency and performance. Precise control of steering angles allows the first virtual character to more accurately perform various actions and interactions, thereby enhancing the precision and controllability of action operations.

[0217] The above content introduces the role of the decision network in multi-level action selection, including the coarse-grained and fine-grained sub-action selection process. Through multi-level action selection, it helps to guide the execution of each action step at both coarse and fine granularity, thereby improving the execution accuracy and efficiency of the overall action. By gradually narrowing the scope of action, the selection and execution method of the action can be adjusted and optimized according to the changes in the virtual environment and task requirements, enhancing the adaptability and flexibility of the object sub-action determination. The action selection of each step is based on the results of the previous step and the detailed action description, reducing the decision-making errors caused by incomplete information or misunderstanding, and improving the reliability and stability of the object sub-action determination.

[0218] In one possible implementation, to prevent the avatar from frequently performing a certain action, which is inconsistent with human behavior logic, a cooldown period can be set for some sub-actions. If the cooldown period of a sub-action has not yet completed, the sub-action is masked. This prevents the avatar from repeatedly performing the same action, which can cause twitching.

[0219] In an embodiment of the present application, action masking is performed on the sub-actions in the action type to be decided based on three different dependency restriction situations and the determined object sub-actions, thereby improving the action decision efficiency in high-dimensional complex action space and saving computing resources to a certain extent.

[0220] In some embodiments, the character state information includes a variety of scalar information describing the current situation, including information about the state of the first virtual character and information about the first virtual character's interaction with the environment. Essentially, environmental perception information is derived from the first virtual character's perception of the surrounding virtual environment, depicting the relative characteristics of the first virtual character relative to the virtual environment. Therefore, training the action decision model in a sufficiently rich virtual environment ensures that the relative characteristics captured by the action decision model are sufficiently rich and diverse, thereby ensuring the action decision model's generalized ability to perceive environmental perception.

[0221] However, the character status information may contain some specific features that are strongly associated with a specific map scene, so the computer device will perform further generalization processing based on the features.

[0222] Optionally, the computer device generalizes the absolute state information included in the character state information to obtain relative state information, where the relative state information is state information of the first virtual character relative to other virtual characters or obstacles.

[0223] Optionally, generalization processing can be achieved by replacing absolute information with relative information. For example, the absolute position information or absolute position orientation of the first virtual character, other virtual characters, and cover points can be replaced with the relative position, i.e., position orientation, between virtual characters and virtual characters, and between virtual characters and cover points / obstacles.

[0224] In the embodiment of the present application, by generalizing the input information of the action decision model and combining it with a variety of training scenarios, an action decision model with good generalization can be trained, so that the action decision model can be easily deployed in various map scenarios; among them, relative state information is more helpful for the first virtual character to make decisions and actions in a dynamic environment. The dependence on the absolute coordinate system is eliminated through generalization processing, so that the virtual character can respond more flexibly to changes in the surrounding environment, improve the perception and decision-making ability of the first virtual character, and enhance the adaptability and intelligent performance in a dynamic environment, effectively promoting the behavior optimization and real-time response of the virtual character in complex scenarios.

[0225] Please refer to Figure 14, which shows a schematic diagram of an action decision model provided by an exemplary embodiment of the present application. It includes an encoding network, a decision network, an action mask network, and a value network. The computer device inputs the acquired character state information (including vector interaction features) and environmental perception information into the encoding network of the action decision model for encoding and splicing to obtain splicing features. The splicing features are then extracted through the LSTM network to obtain fused features. The fused features are input into the decision network, which includes a fully connected layer and an activation function ReLU. The decision network determines the character action based on the received fused features. In the process of making action decisions, the sub-actions are masked based on the action mask network, and finally the object sub-actions are determined through action sampling.

[0226] Furthermore, during the training process, the action decision model also includes a value network. This network evaluates the value of a specific sub-action and adjusts the action decision model based on the estimated value. Furthermore, partial game information can be fed into the value network to enable it to estimate the value of the sub-action based on the current game situation.

[0227] Before applying the action decision model, the model should be trained in a training game. The following will illustrate the training process of the action decision model through an exemplary embodiment.

[0228] Please refer to FIG15 , which shows a flowchart of an action decision model training process provided by an exemplary embodiment of the present application. The process includes the following steps:

[0229] Step 1501: Obtain sample character state information of a first virtual character and sample environment perception information of the first virtual character to the current virtual environment.

[0230] Among them, sample environmental perception information of different modalities in the sample environmental perception information has different information dimensions.

[0231] The specific implementation of this step can refer to the process of obtaining the character status information and the environment perception information in the above step 201, which will not be described in detail in this embodiment.

[0232] Step 1502 : Feature encoding and fusion of the sample character state information and the sample environment perception information are performed through the encoding network of the action decision model to obtain sample fusion features.

[0233] Among them, the encoding network features the sample role state information and sample environment perception information of different modalities through different encoding methods.

[0234] The specific implementation of this step can refer to the process of obtaining fusion coding based on the character status information and the environment perception information in the above step 202, which will not be described in detail in this embodiment.

[0235] Step 1503: Based on the sample fusion features, the action decision model is trained by reinforcement learning.

[0236] Steps 1501 through 1503 describe training a decision model based on sample character state information and sample environmental perception information. The model training process corresponds to applying the model to action decisions. This training process helps improve the robustness of the decision model and enhances its accuracy when making action decisions.

[0237] The process of training the action decision model based on reinforcement learning includes the following.

[0238] Step 1503a: Input the sample fusion features into the decision network in the action decision model to obtain the estimated character action.

[0239] In fact, the process of training the action decision model is the process of strengthening the training of the decision network in the action model.

[0240] The specific implementation of this process can refer to the above embodiment for determining the content of the character action, and this embodiment will not be described in detail here.

[0241] Step 1503b: Control the first virtual character to execute the estimated character action, and obtain the estimated action execution result of the first virtual character.

[0242] The estimated action execution result includes a change in character state information of the first virtual character after the first virtual character executes the estimated character action and a change in environment perception information perceived by the first virtual character.

[0243] Step 1503c: Determine an estimated action execution reward based on the estimated action execution result of the first virtual character.

[0244] During the training process, certain tasks will be set for the first virtual character. Therefore, in order to avoid the first virtual character taking extreme operations to complete the task, resulting in a low degree of anthropomorphism of the first virtual character, in addition to the execution reward of the character action based on the task progress, the execution reward of the estimated character action will also be determined based on the degree of anthropomorphism of the estimated character action.

[0245] First, the computer device will execute the weight coefficient ratio of each reward and punishment item, which is divided into a dominant reward (i.e., a reward for the progress of the preset task) and an auxiliary anthropomorphic reward. In the early stage of training the action decision model, the weight coefficient of the dominant reward is increased and the weight coefficient of the auxiliary anthropomorphic reward is reduced, so that the action decision model prioritizes learning for the purpose of completing the task. After multiple rounds of training, the strength of the action decision model continues to increase, and it is necessary to adjust the weight coefficient of the auxiliary anthropomorphic reward based on the performance of the action decision model and the trend of changes in the corresponding anthropomorphic index. Among them, the anthropomorphic index refers to the degree of match between the probability of the first virtual character performing a specific action and the probability of the real player performing the characteristic action. For example, if the first virtual character is unable to reasonably use cover to engage in pulling battles in the game, it can be determined that the weight of the dominant reward is too large, and the weight of the dominant reward is reduced to ensure that the action decision model can determine the anthropomorphic character action based on a certain strength.

[0246] In an embodiment of the present application, the estimated action execution reward mainly includes at least one of an attribute reward, a game victory reward, a game failure reward, and a task reward.

[0247] The attribute reward mainly includes the health attribute reward of the first virtual character. When the attribute value of the first virtual character decreases, the attribute reward is determined as the estimated action execution reward. For example, after the first virtual character performs the estimated character action, the health attribute of the first virtual character decreases, and it is determined that the character action obtains a negative attribute reward. Alternatively, when the attribute value of the first virtual character increases, the attribute reward is determined as the estimated action execution reward. For example, after the first virtual character performs the estimated character action, the health attribute of the first virtual character increases, and it is determined that the character action obtains a positive attribute reward.

[0248] The game failure reward means that when the first virtual character fails in the game, a huge reverse reward (ie, a penalty) will be obtained. When the first virtual character fails in the game, the game failure reward is determined as an estimated action execution reward.

[0249] The reward for a successful game is the opposite of the reward for a failed game. When the first virtual character succeeds in the game, a huge positive reward will be obtained. When the attribute value of the first virtual character decreases, the attribute reward will be determined as the estimated action execution reward.

[0250] In some embodiments, the first virtual character has a preset task in the game, such as guarding a treasure chest, avoiding pursuit, or reaching a designated location, etc. Therefore, if the first virtual character succeeds in the task in the training game, the task reward is determined as the estimated action reward.

[0251] Furthermore, to enable the action decision model to make anthropomorphic action decisions, an anthropomorphic reward is also included in the estimated action reward. The computer device determines anthropomorphic attribute values ​​for at least two estimated character actions based on the estimated action execution results of the first virtual character performing at least two estimated character actions in a training game. The anthropomorphic attribute value may include the number of times a specific action is performed, the movement positions of at least two consecutively determined estimated character actions, and the like. If the anthropomorphic attribute value is below an anthropomorphic attribute threshold, the anthropomorphic reward is determined as the estimated action execution reward.

[0252] For illustration, please refer to Table 3, which shows the estimated action execution reward provided by an exemplary embodiment of the present application.

[0253] Table 3

[0254] In the table above, some estimated action rewards require judgment based on multiple consecutive estimated character actions. For example, a reasonable sidestep is an anthropomorphic reward. Virtual characters controlled by real players generally do not sidestep multiple times in a short period of time, or sidestep when there is no cover around. Therefore, if the anthropomorphic attribute value of the sidestep action (i.e., the number of sidestep actions) exceeds the anthropomorphic attribute threshold among the multiple determined character actions, it is determined to be an unreasonable sidestep and a negative reward is applied based on the action decision model.

[0255] Table 3 includes rewards for dispersed movement, rewards for using cover, rewards for ambushing in the dark, and rewards for reasonable actions (reasonable sideways movement, reasonable squatting, reasonable still walking, etc.).

[0256] 1. Movement Dispersion Bonus: This reward is awarded for reducing the closeness of the first avatar's movements over multiple consecutive times while executing the predicted action. This means the first avatar won't wander around in the same place for multiple times, and can move more dispersedly across large maps, exploring different locations.

[0257] Optionally, the estimated action execution result includes the position point after the first virtual character performs the first n+1 actions. Based on the position point after the first virtual character performs the n+1th estimated character action, the computer device determines a first closeness between the position point after performing the n+1th action and the position point after performing the previous n actions, and determines a second closeness between the position point after performing the nth action and the position point after performing the previous n-1 actions. When the first closeness is less than the second closeness, the movement dispersion reward is determined as the estimated action execution reward. When training the action decision model, it may cause the first virtual character to be trapped in a local area and go in circles. Therefore, based on the close centrality algorithm, the first virtual character can be guided to perform the estimated character action determined by the action decision model, so that the movement can be dispersed enough and the movement route can cover the entire map as much as possible, so as to train the movement ability of the first virtual character.

[0258] Optionally, when the first closeness between the position point after executing the n+1th action and the position point after executing the previous n actions is less than the second closeness between the position point after executing the nth action and the position point after executing the previous n-1 actions, it is determined that a forward movement dispersion reward is obtained. Alternatively, when the closeness between the position point after executing the n+1th action and the position point after executing the previous n actions is greater than the closeness between the position point after executing the nth action and the position point after executing the previous n-1 actions, it is determined that a reverse movement dispersion reward is obtained. This guides the first virtual character to stay as far away from the most recently visited historical location as possible, avoiding the situation where the virtual character wanders in place.

[0259] The above process introduces the use of position dispersion rewards as an estimated action execution reward. By rewarding the first avatar's position changes over different time periods, it encourages more diverse tactical strategies and positioning methods, increasing the tactical depth and challenge of the game or simulation. The position dispersion reward also makes the first avatar more difficult to predict in combat or competitive environments, thereby improving the first avatar's survivability and escape capabilities. Furthermore, by considering the closeness of position points, it helps to better optimize the action decision-making model, enabling it to more intelligently select and execute actions to maximize rewards and improve overall performance.

[0260] Second, the cover utilization bonus is the reward for the first avatar being adjacent to cover when performing the predicted action. In the presence of other avatars, the first avatar is expected to stay close to cover as much as possible to avoid being attacked by other avatars.

[0261] Optionally, the estimated action execution result includes sample environment perception information after the first virtual character performs the estimated character action, and the sample environment perception information includes a two-dimensional depth map, which is used to represent the obstruction of the first virtual character by obstacles in the direction of the first virtual character.

[0262] The computer device divides the two-dimensional depth map into regions to obtain at least two depth regions. A first occlusion ratio is determined for the first depth region, and a second occlusion ratio is determined for at least two second depth regions adjacent to the first depth region, where the first depth region is located at the center of the first virtual character's field of view. A cover utilization reward is determined based on a minimum difference between the first and second occlusion ratios, wherein the degree of the cover utilization reward is positively correlated with the minimum difference, and the cover utilization reward is determined as the estimated action execution reward.

[0263] The above content describes how to divide a two-dimensional depth map into regions and determine cover utilization rewards based on the cover ratios of these regions. By evaluating and rewarding cover utilization, the first avatar is encouraged to choose more optimal cover positions during combat, optimizing their defensive or offensive capabilities. Taking into account the differences in cover ratios between depth regions, the avatar is encouraged to choose positions that offer better vision and protection, thereby enhancing their tactical awareness and visual strategy. This not only adds tactical depth to the game, but also enhances the player's combat experience, making the game more realistic and challenging.

[0264] In the embodiments of the present application, the determination of cover utilization rewards is achieved by reshaping the depth map matrix. In a game scene, a cover is an area that provides cover for the first virtual character during combat. Multiple covers exist in the virtual environment, and the different covers differ in at least one of their positions, orientations, and shapes. A two-dimensional depth map can characterize the depth of obstacles facing the first virtual character. Therefore, the pixel distribution area of ​​the cover can be determined based on the pixel value distribution of the two-dimensional depth map.

[0265] Please refer to Figure 16, which shows a schematic diagram of regional division of a depth map provided by an exemplary embodiment of the present application. The depth map is divided according to the numerical direction, resulting in 19 regions 1-19. The depth map is vertically divided into several columns, and the brightness rate of each column is calculated. The larger the brightness rate, the brighter the front, the less likely there is an obstacle blocking it, and the lower the obscuration rate. In the application, it is expected that the first virtual character can choose an area with cover and can attack other virtual characters. Therefore, it is expected that the brightness rate of the middle column of the depth map is the smallest. Since the middle column reflects the field of view information directly in front of the first virtual character, the smaller the brightness rate, the greater the obscuration rate, and the more likely there is a cover area directly in front. In addition, it is expected that the first virtual character can be at the edge of the cover, that is, able to attack other virtual characters by moving sideways, etc. Therefore, the absolute value of the difference between the brightness rate of the middle column and the larger brightness rate of the two columns to its left and right should be as large as possible, that is, the minimum value of the difference between the brightness rate of the middle column and the brightness rate (obscuration rate) of the two columns to its left and right is as large as possible. Corresponding to FIG16 , it is desired that the masking rate of the 10th column is as large as possible, and the masking rate of an area in the 9th and 11th columns is as small as possible.

[0266] Optional, the brightness ratio v corresponding to the depth area j It can be calculated by the following formula:

[0267] where v j The ratio of the actual pixel in each depth region to the theoretical maximum pixel sum in each column, and the masking rate of the depth region is 1-v j The bunker utilization bonus can be calculated as follows: r = α·(1-v 10 )+β·|1-v 10 -min(1-v9,1-v 11 )|

[0268] Here, r is the cover exploitation reward, and α and β are the reward hyperparameters.

[0269] It should be noted that the method of dividing the two-dimensional depth map into regions can also be based on the horizontal direction, or the number of divided regions can be adjusted based on the virtual environment. In the case of a more complex virtual environment, more depth regions can be used to perform more fine-grained division of the two-dimensional depth map, thereby obtaining a more refined cover effect. In the case of a large map scene and a relatively simple control structure, a coarse-grained depth region division can be performed, which is not limited in this embodiment.

[0270] 3. Ambush Bonuses are also known as Visibility Bonuses. Visibility Bonuses are rewards awarded when a virtual character executes a predicted action and is effectively positioned to attack other virtual characters. When other virtual characters are present, it's expected that they will be visible to the first virtual character. However, if the first virtual character is not visible to the other virtual characters, this facilitates an attack.

[0271] Optionally, the estimated action execution result includes a mutual visibility relationship between the first virtual character and the other virtual characters, where the mutual visibility relationship is used to represent the visibility of the first virtual character and the other virtual characters relative to each other.

[0272] The computer device determines the estimated action execution reward as a positive visibility reward when the mutual visibility relationship indicates that the first virtual character is not within the visibility range of the other virtual characters and the other virtual characters are within the visibility range of the first virtual character.

[0273] When the mutual visibility relationship indicates that the first virtual character is within the visibility range of the other virtual character, and the other virtual character is not within the visibility range of the first virtual character, the estimated action execution reward is determined to be a reverse visibility reward.

[0274] That is, when the first virtual character is in a favorable attacking position compared to other virtual characters, a positive visibility reward is determined to be obtained. Conversely, when the first virtual character is in a disadvantageous attacking position compared to other virtual characters, a negative visibility reward is determined to be obtained.

[0275] Optionally, the mutual visibility relationship can be obtained by ray detection. Taking the eye position of the first virtual character as the starting point, environmental perception rays are emitted to various parts of other virtual characters in the direction of the first virtual character. According to the reflection of the environmental perception rays, the visibility of various parts of other virtual characters in the virtual environment is determined, and the visibility is used to characterize whether various parts of other virtual characters are blocked by cover. When the visibility indicates that various parts of other virtual characters are blocked by obstacles, it is determined that the other virtual characters are not in the visible range of the first virtual character. Similarly, taking the eye position of other virtual characters as the starting point, environmental perception rays are emitted to various parts of the first virtual character in the direction of other virtual characters. According to the reflection of the environmental perception rays, the visibility of various parts of the first virtual character is determined. When the visibility indicates that various parts of the first virtual character are blocked by obstacles, it is determined that the first virtual character is not in the visible range of other virtual characters.

[0276] The above describes the process of determining the directionality of estimated action execution rewards based on the mutual visibility between virtual characters. Positive visibility rewards encourage virtual characters to take actions without being noticed by other characters. This can be used to enhance strategies such as stealth, concealment, and tactical attacks, thereby increasing combat depth and tactical options within a game or simulation. Negative visibility rewards encourage virtual characters to take actions upon being spotted by other characters. This promotes quick reactions, avoidance of exposure, and the implementation of defensive strategies, helping to enhance the survivability and gaming experience of the virtual characters. This effectively enriches the behavioral options available to virtual characters within a game or simulation, improving the overall gaming experience and engagement.

[0277] 4. Reasonable Action Rewards refer to rewards given to the first avatar when it performs the predicted actions, consistent with the actual player's operating logic. During model training, we expect the first avatar's behavior to be as human-like as possible, performing actions that are more consistent with human logic.

[0278] Before determining the estimated character action, the computer device determines an ideal action for the first virtual character based on the sample character state information and the sample environmental perception information of the first virtual character. For example, if the environmental perception information indicates that there is low cover in front of the first virtual character, the ideal action for the first virtual character may be a squat. For another example, if the environmental perception information indicates that there is another virtual character to the right of the first virtual character, the ideal action may be a turn to the right.

[0279] Subsequently, when the estimated character action corresponding to the action decision model is consistent with the ideal action, the reasonable action reward is determined as the estimated action execution reward.

[0280] The above describes the use of determining ideal actions based on virtual character state information and environmental perception information, and using this information to determine rewards for appropriate actions. By combining character state and environmental information to calculate ideal actions, virtual characters can make more intelligent and context-sensitive action choices, improving the performance and intelligence of the overall game or simulation system. It also helps players interact with virtual characters more intuitively, as the virtual characters exhibit intelligence and realism that are closer to real-world behavior. By rewarding appropriate actions, compassion also effectively promotes the efficient use of resources and the effective execution of tasks, thereby optimizing action execution efficiency and performance.

[0281] In some embodiments, environmental perception rays are emitted in multiple directions with the first virtual character as the starting point, and different environmental perception rays are located at the same horizontal height. When the environmental perception rays collide with the surface of an obstacle, they are reflected along the normal direction of the environmental perception rays; when the reflection of the environmental perception rays indicates that there is a first environmental perception ray whose projection distance is less than a distance threshold, the emission direction of the environmental perception ray adjacent to the first environmental perception ray and whose projection distance is greater than the distance threshold is determined as the ideal turning direction, and the projection distance refers to the distance between the environmental perception ray from the starting point to the reflection point; when the turning direction indicated by the turning sub-action is consistent with the ideal turning direction, the reasonable action reward is determined as the estimated action execution reward.

[0282] Among them, by using environmental perception rays to calculate the ideal turning direction, virtual characters can more effectively avoid obstacles and choose the best path, thereby optimizing their movement and navigation capabilities. By simulating the behavior of objects reflecting light in the real world, the virtual environment is made more realistic and challenging. By rewarding reasonable actions, it not only improves the operating efficiency of virtual characters, but also optimizes the response speed and intelligence level in processing complex scenes and dynamic environments, which helps to improve the efficiency of human-computer interaction.

[0283] The above describes the implementation of estimated action execution rewards. By leveraging at least one of the four aforementioned rewards, different reward mechanisms can not only optimize the first avatar's behavioral decisions but also enhance performance and player experience within the game or simulation environment. By designing and implementing rewards for rational movement, cover utilization, visibility, and positioning dispersion, we can more effectively simulate real-world tactical and strategic choices, improving the intelligence and realism of the first avatar.

[0284] Schematically, please refer to FIG17, which shows that a turning sub-action exists in the character action provided by an exemplary embodiment of the present application. The computer device emits environmental perception rays in multiple directions with the first virtual character as the starting point, and different environmental perception rays are located at the same horizontal height. The computer device determines the distribution of surrounding obstacles based on the reflection of the environmental perception rays. In FIG17, when there is a shortest environmental perception ray whose length is less than the distance threshold, it means that the first virtual character is close to the distance between the obstacles, and it can be determined that a turn should be made at this time. The environmental perception rays spread to the left and right with the shortest ray as the reference, and the computer device determines that the ray direction (the counterclockwise direction of the first virtual character) whose ray length exceeds the distance threshold earlier is the ideal turning direction of the first virtual character.

[0285] If the steering direction determined by the action decision model is consistent with the ideal steering direction, the reasonable action reward is determined as the estimated action execution reward. If the steering direction determined by the action decision model is inconsistent with the ideal steering direction, the first virtual character may become stuck in the wall, and a certain reverse reward is given.

[0286] Step 1503d: Update the model parameters of the action decision model based on the estimated action execution reward.

[0287] The process of updating the action decision model based on the estimated action execution reward is to modify the model parameters to obtain a larger estimated action execution reward. This allows the action decision model to achieve the most human-like action possible while performing the preset task.

[0288] In an embodiment of the present application, during the training of the action decision-making model, adding auxiliary anthropomorphic action rewards can meet the dual requirements of model strength and anthropomorphism. While accurately predicting and controlling the character behavior of the first virtual character, the anthropomorphism of the first virtual character is greatly improved by combining the action masking logic. Since the rewards of the actual execution results can also be immediately fed back to the action decision-making model, the decision-making strategy of the model is optimized through the feedback of the rewards, so that it is gradually improved and optimized, and it is more in line with the behavioral logic of real players while better adapting to various scenarios and challenges.

[0289] In the embodiment of the present application, a first virtual character is in a game with at least one other virtual character, and the other virtual character executes an action output by the second action decision model. In some scenarios, the task of the other virtual character corresponds to the task of the first virtual character. For example, the first virtual character's task is to guard a treasure chest, while the task of the other virtual character may be to capture the treasure chest.

[0290] Other virtual characters, serving as opponent virtual characters, also need to iterate the second action decision model after training the first action decision model corresponding to the first virtual character to obtain second action decision models of different strengths.

[0291] Optionally, when a first action decision model corresponding to the first virtual character completes one round of training, a second action decision model is trained based on the trained first action decision model, and the second action decision model after one round of training is stored in the opponent model pool. The opponent model pool contains second action decision models that have completed at least one round of training.

[0292] During the next round of training for the first decision model, a previously trained second decision model is sampled from the opponent model pool according to a certain rule as an opponent for the next round of training. The computer device selects a trained second action decision model from the opponent model pool and constructs a new game. The new game includes the first virtual character executing the predicted character action output by the first action decision model and other virtual characters executing the action output by the second action decision model. The first action decision model is then trained in the new game.

[0293] Indicatively, the specific process is as follows:

[0294] (1) Based on the client's built-in behavior tree, the opponent virtual character is controlled and the first action decision model is trained for round 0; (2) The model trained in round 0 is stored in the evaluation model pool for evaluation; (3) Based on the first action decision model that has completed round 0 training, the second action decision model is trained for round 0 and added to the opponent model pool; (4) A previously trained second decision model is selected from the opponent model pool as the opponent and the first action decision model is trained for round K; (5) The first action decision model trained in round K is stored in the evaluation model pool for evaluation; (6) The second action decision model is trained for round K based on the first action decision model trained in round K and stored in the opponent model pool; (7) Steps (4) to (6) are repeated until the evaluation result of the first action decision model satisfies the performance evaluation result, and the first action decision model is determined to have completed training.

[0295] The above content describes the process of training the action decision model and building a game for a virtual character. By using the training experience of the first model to accelerate the learning process of the second model, multiple decision models can be optimized more quickly; storing multiple trained models in the opponent model pool helps to build more challenging and varied games, and improve the virtual character's adversarial learning ability and adaptability; by training and adjusting the first action decision model in actual games, the character's behavioral strategy and intelligence level can be continuously optimized to cope with complex and dynamic game or simulation environments.

[0296] In one possible implementation, in order to make the first action decision model have a certain strength and anthropomorphism, the following sparring behaviors can be added to the opponent virtual character:

[0297] 1. Creating an opponent virtual character that sprints and chases throughout the game is helpful in training the first virtual character's ability to explore the entire map.

[0298] 2. Set the spawn point of the opponent virtual character to a random spawn point. If the spawn distance between the opponent virtual character and the first virtual character is close, set a cool-down period for the opponent virtual character to avoid the first virtual character being instantly defeated at the beginning of the training process.

[0299] 3. When the distance between the opponent's virtual character and the first virtual character is less than the safe distance, the first virtual character is determined to have failed the game. This can train the first virtual character to maintain a certain distance from the opponent's virtual character and prevent the first virtual character from being stuck by the terrain.

[0300] 4. Make the opponent's virtual character move in front of the first virtual character with a small probability, thereby simulating the scene of the first virtual character in the real player dimension to enhance the strength of the first virtual character.

[0301] In an embodiment of the present application, by using a second action decision model corresponding to the sparring opponent virtual character and a first decision model corresponding to the first virtual character, the strength of the first decision model is improved, so that the first decision model can have stable action decision-making capabilities.

[0302] In this embodiment of the application, judging whether the action decision model has completed training requires combining the game index and the anthropomorphism index. The game index is used to judge the strength of the action decision model, and the anthropomorphism index is used to judge the degree of anthropomorphism of the action decision model.

[0303] Please refer to Table 4, which shows the game indicators and anthropomorphic indicators provided by an exemplary embodiment of the present application.

[0304] Table 4

[0305] Among them, when determining the anthropomorphic index, the average anthropomorphic index is determined based on a large amount of game data.

[0306] After completing the i-th round of training for the action decision model, the computer device obtains game data from the i-th round of training games. Based on the game data from at least two training rounds, a game index and an anthropomorphic index are determined. The anthropomorphic index refers to the percentage of specific actions actually executed in the game data from at least two training games.

[0307] When the game indicators meet the training completion standards and the anthropomorphic indicators match the action execution ratio of real players performing specific actions in the game, the action decision model is determined to have completed training.

[0308] In the embodiment of the present application, game indicators and anthropomorphism indicators are jointly used as evaluation criteria for the action decision model, so that the trained action decision model can take into account both the strength and the degree of anthropomorphism of the action model; not only does it improve the performance and practicality of the action decision model, but it can also ensure its excellent performance in a variety of game scenarios, meeting the expectations and needs of real players.

[0309] Please refer to FIG18 , which shows a schematic structural diagram of an action decision device for a virtual character provided by an exemplary embodiment of the present application. The device includes the following structure.

[0310] An acquisition module 1801 is configured to acquire character state information of a first virtual character and environmental perception information of the first virtual character regarding a current virtual environment, wherein the environmental perception information includes perception information in multiple modalities, each of the multiple modalities representing an information dimension.

[0311] The encoding module 1802 is configured to fuse the encoding results corresponding to the character state information and the environmental perception information to obtain a fusion feature, wherein the encoding result is a result of feature encoding the character state information and the plurality of perception information using different encoding methods;

[0312] A decision module 1803 is configured to make an action decision for the first virtual character based on the fusion feature to obtain a character action;

[0313] The control module 1804 is configured to control the first virtual character to perform the character action.

[0314] Optionally, the multimodal environment perception information includes first modality perception information, second modality perception information, and third modality perception information;

[0315] The acquisition module 1801 is used to: acquire first modal perception information through ray detection based on the character position of the first virtual character in the virtual environment, the first modal perception information being a one-dimensional vector and being used to characterize the depth of obstacles at the same horizontal height around the first virtual character; acquire second modal perception information through ray detection based on the character position of the first virtual character in the virtual environment and the orientation of the first virtual character, the second modal perception information being a two-dimensional depth map in the orientation direction of the first virtual character and being used to characterize the depth of obstacles in the orientation direction of the first virtual character; acquire third modal perception information of the virtual environment within a preset range of the first virtual character through ray detection based on the character position of the first virtual character in the virtual environment, the third modal perception information being a three-dimensional occupancy grid map and being used to characterize the spatial distribution of obstacles within the preset range.

[0316] Optionally, the acquisition module 1801 is used to: emit environmental perception rays in multiple directions with at least two heights of the character position of the first virtual character as starting points; wherein, when the environmental perception rays collide with the surface of an obstacle, they are reflected along the normal direction of the environmental perception rays, and the emission directions of different environmental perception rays emitted at the same height are at the same horizontal height; and generate the first modal perception information based on the reflection of the environmental perception rays.

[0317] Optionally, the acquisition module 1801 is used to: emit an environmental perception ray in the direction of the first virtual character with the first virtual character as the starting point, wherein when the environmental perception ray collides with the surface of an obstacle, it is reflected along the normal direction of the environmental perception ray; and generate the second modal perception information based on the reflection of the environmental perception ray.

[0318] Optionally, the acquisition module 1801 is configured to: divide the virtual environment within a preset range of the first virtual character using an occupation grid, wherein the occupation grid is composed of multiple cubes of equal volume; emit environmental perception rays from at least two starting points in the virtual environment into the occupation grid, and generate the third modal perception information based on reflections of the environmental perception rays. Optionally, perform action decisions for the first virtual character using an action decision model, wherein the action decision model includes an encoding network and a decision network;

[0319] The encoding module 1802 is used to: input the character state information and the environmental perception information into the action decision model, encode the character state information and the environmental perception information through the encoding network, and obtain a state information encoding result corresponding to the character state information and a perception information encoding result corresponding to the environmental perception information; splice the state information encoding result and the perception information encoding result to obtain a spliced ​​feature code; extract the spliced ​​feature code through the encoding network feature to obtain the fusion feature.

[0320] Optionally, the encoding network includes a state information encoder and a perception information encoder, the perception information encoder includes convolution kernels of multiple dimensions, and the convolution kernels of multiple dimensions are respectively used to perform feature encoding on the perception information of multiple modalities;

[0321] The encoding module 1802 is used to: encode the character state information through the state information encoder to obtain the state information encoding result corresponding to the character state information; encode the first modal perception information through the one-dimensional convolution kernel in the perception information encoder to obtain the first perception information encoding result; encode the second modal perception information through the two-dimensional convolution kernel in the perception information encoder to obtain the second perception information encoding result; encode the third modal perception information through the three-dimensional convolution kernel in the perception information encoder to obtain the third perception information encoding result. The first perception information encoding result, the second perception information encoding result and the third perception information encoding result constitute the perception information encoding result.

[0322] Optionally, the character state information includes absolute state information, where the absolute state information is used to represent state information of the first virtual character in the virtual environment that is independent of other virtual characters or obstacles;

[0323] The device further includes a generalization module configured to generalize the absolute state information contained in the character state information to obtain relative state information, where the relative state information is state information of the first virtual character relative to other virtual characters or obstacles.

[0324] Optionally, the decision network includes n action output heads, the n action output heads are used to serially output n object sub-actions, different action output heads correspond to different action types, and the n action output heads are serially connected based on a dependency relationship between the action types, the dependency relationship is used to characterize the dependency restrictions between sub-actions under different action types, the n object sub-actions constitute the character action, and n is a positive integer greater than 2;

[0325] The decision module 1803 includes: inputting the fused features into the first action output head in the decision network, determining the first object sub-action from the sub-actions of the first action type through the first action output head in the decision network; inputting the embedded coding vector corresponding to the determined first object sub-action to the i-1th object sub-action, as well as the fused features, into the i-th action output head, determining the i-th object sub-action from the sub-actions of the i-th action type through the i-th action output head, where i is a positive integer greater than 1. Optionally, among the n action output heads, there are at least two action output heads corresponding to the same action type, there is the i-th action output head corresponding to a coarse-grained sub-action, and there is the j-th action output head corresponding to a fine-grained sub-action, where i is less than j, and j is a positive integer greater than 2.

[0326] The decision module 1803 includes: determining the i-th object sub-action from the coarse-grained sub-actions through the i-th action output head in the decision network; based on the fusion vector and the embedded coding vector corresponding to the i-th object sub-action, determining the j-th object sub-action from the fine-grained sub-actions through the j-th action output head, and the action range indicated by the j-th object sub-action is smaller than the action range indicated by the i-th character action.

[0327] Optionally, the device further includes:

[0328] an action masking module, configured to: perform action masking on the sub-actions in the i-th action type corresponding to the i-th action output head based on the determined dependency constraints between the first object sub-action to the i-1-th object sub-action, and between different sub-actions in different action types, wherein at least two sub-actions having a dependency constraint relationship indicated by the dependency constraint cannot be executed simultaneously, and the action mask is used to shield unreasonable or invalid sub-actions;

[0329] The decision module 1803 includes: determining the i-th object sub-action from the unmasked sub-actions in the i-th action type through the i-th action output head.

[0330] Optionally, the device further includes:

[0331] a determination module configured to emit environment perception rays, starting from the eye position of the first virtual character, toward various parts of other virtual characters in the direction of the first virtual character; and determine, based on reflections of the environment perception rays, visibility of various parts of the other virtual characters in the virtual environment, wherein the visibility is used to indicate whether various parts of the other virtual characters are obscured by cover.

[0332] The masking module is configured to perform action masking on the sub-actions in the character action type based on the visual conditions of the various parts of the other virtual characters, wherein the object sub-actions have the dependency restriction relationship with the visual conditions of the various parts of the other virtual characters;

[0333] The decision module 1803 includes determining an object sub-action from unmasked sub-actions in the character action type through a character action output head.

[0334] Optionally, among the n action output heads, there is a first steering action output head corresponding to a coarse-grained steering action, and there is a second steering action output head corresponding to a fine-grained steering action, the first steering action output head is used to determine a first steering angle, and the second steering action output head is used to determine a second steering angle;

[0335] The determining module is further configured to: determine an ideal turning angle based on the orientation of the first virtual character, the position coordinates of the first virtual character, and the position coordinates of other virtual characters, and determine an aiming distance between the first virtual character and the other virtual characters; and determine an effective aiming range based on the aiming distance and the widths of the other virtual characters;

[0336] The action masking module is configured to: determine an angle difference between a first steering angle determined by the first steering action output head and the ideal steering angle based on the first steering angle determined by the first steering action output head; determine an effective fine-grained steering angle based on the angle difference and the effective aiming range; and perform action masking on steering angles in the coarse-grained steering action that do not belong to the effective fine-grained steering angles;

[0337] The decision module 1803 is configured to determine the second steering angle from the valid fine-grained steering angles that are not masked in the fine-grained steering action through the second steering action output head.

[0338] Optionally, the device further includes:

[0339] a training module configured to obtain sample character state information of the first virtual character and sample environmental perception information of the first virtual character regarding the current virtual environment, wherein the sample environmental perception information includes sample perception information of multiple modalities, each of the multiple modalities representing an information dimension;

[0340] The training module is further configured to fuse the sample encoding results corresponding to the sample character state information and the sample environmental perception information respectively through the encoding network of the action decision model to obtain a sample fusion feature, wherein the encoding network performs feature encoding on the sample character state information and the sample environmental perception information of different modalities through different encoding methods;

[0341] The training module is further used to train the action decision model through reinforcement learning based on the sample fusion features.

[0342] Optionally, the training module is used to: input the sample fusion features into the decision network in the action decision model to obtain an estimated character action; control the first virtual character to perform the estimated character action to obtain an estimated action execution result of the first virtual character; determine an estimated action execution reward based on the estimated action execution result of the first virtual character; and update the model parameters of the action decision model based on the estimated action execution reward.

[0343] Optionally, the estimated action execution reward includes at least one of a reasonable action reward, a cover utilization reward, a visibility reward, and a movement dispersion reward;

[0344] The reasonable action reward refers to the reward for the first virtual character performing the estimated character action in accordance with the real player's operation logic; the cover utilization reward refers to the reward for the first virtual character being adjacent to the cover when performing the estimated character action; the visibility reward refers to the reward for the virtual character being in an effective attack position for other virtual characters when performing the estimated character action; the movement dispersion reward refers to the reward for the first virtual character performing the estimated character action for multiple consecutive movements with reduced density.

[0345] Optionally, the estimated action execution result includes a position point after the first virtual character performs the previous n+1 actions;

[0346] The training module is configured to determine, based on the position point of the first virtual character after performing the estimated character action for the n+1th time, a first closeness between the position point after performing the n+1th action and the position point after performing the previous n actions, and to determine a second closeness between the position point after performing the nth action and the position point after performing the previous n-1 actions;

[0347] When the first density is less than the second density, the movement dispersion reward is determined as the estimated action execution reward.

[0348] Optionally, the estimated action execution result includes the sample environment perception information after the first virtual character performs the estimated character action, the sample environment perception information includes a two-dimensional depth map, and the two-dimensional depth map is used to represent the obstruction of the first virtual character by obstacles in the direction in which the first virtual character is facing;

[0349] The training module is used to: divide the two-dimensional depth map into regions to obtain at least two depth regions; determine a first masking rate of a first depth region and a second masking rate of at least two second depth regions adjacent to the first depth region, wherein the first depth region is located at the center of the field of view of the first virtual character; determine the cover utilization reward based on a minimum difference between the first masking rate and the second masking rate, wherein the reward level of the cover utilization reward is positively correlated with the minimum difference; and determine the cover utilization reward as the estimated action execution reward.

[0350] The estimated action execution result includes a mutual visibility relationship between the first virtual character and other virtual characters, wherein the mutual visibility relationship is used to represent the visibility of the first virtual character and other virtual characters relative to each other;

[0351] The training module is used to: determine the estimated action execution reward as the positive visibility reward when the mutual visibility relationship indicates that the first virtual character is not within the visual range of other virtual characters, and other virtual characters are within the visual range of the first virtual character; and determine the estimated action execution reward as the negative visibility reward when the mutual visibility relationship indicates that the first virtual character is within the visual range of other virtual characters, and other virtual characters are not within the visual range of the first virtual character.

[0352] The training module is also used to determine the ideal action of the first virtual character based on the sample character state information and the sample environment perception information of the first virtual character; when the estimated character action is consistent with the ideal action, the reasonable action reward is determined as the estimated action execution reward.

[0353] Optionally, the character action includes a turning sub-action;

[0354] The training module is used to: emit environmental perception rays in multiple directions with the first virtual character as the starting point, and different environmental perception rays are located at the same horizontal height. When the environmental perception rays collide with the surface of an obstacle, they are reflected along the normal direction of the environmental perception rays; when the reflection of the environmental perception rays indicates that the projection distance of the first environmental perception ray is less than a distance threshold, the emission direction of the environmental perception ray adjacent to the first environmental perception ray and with a projection distance greater than the distance threshold is determined as the ideal turning direction, and the projection distance refers to the distance between the environmental perception ray from the starting point to the reflection point; when the turning direction indicated by the turning sub-action is consistent with the ideal turning direction, the reasonable action reward is determined as the estimated action execution reward.

[0355] Optionally, there is at least one other virtual character in the game in which the first virtual character is playing, and the other virtual character executes the action output by the second action decision model;

[0356] The training module is used to: when the first action decision model corresponding to the first virtual character completes a round of training, train the second action decision model based on the trained first action decision model, and store the second action decision model after completing one round of training in the opponent model pool; select the trained second action decision model from the opponent model pool, and construct a new game, wherein the new game includes the first virtual character that executes the estimated character action output by the first action decision model, and other virtual characters that execute the action output by the second action decision model; and train the first action decision model in the new game.

[0357] Optionally, the training model is used to: obtain the game data of the i-th round of training games when the i-th round of training of the action decision model is completed; determine the game index and the anthropomorphic index based on the game data of at least two rounds of training games, and the anthropomorphic index refers to the actual execution ratio of a specific action in the game data of at least the training games; when the game index reaches the training completion standard and the anthropomorphic index matches the action execution ratio of the specific action performed by real players in the game, determine that the action decision model has completed training.

[0358] In summary, when a first virtual character needs to make an action decision in a virtual environment, the character state information and multimodal environmental perception information of the first virtual character are obtained, and feature encoding and fusion of the character state information and multimodal environmental perception information are then performed, ultimately making an action decision based on the fused features. Acquiring environmental perception information of different information dimensions perceived by the first virtual character when making an action decision helps to obtain a more comprehensive and detailed three-dimensional virtual environment surrounding the first virtual character, thereby enabling more reasonable action decisions. Using different encoding methods to feature encode the character state information and multiple different modal perception information enables the action decision model to more accurately fit various types of information in the current scene. Furthermore, compared to the related art method of using image recognition to obtain environmental features, extracting features from quantified multimodal environmental perception information and then fusing the encoding results of the character state information helps to more fully consider the character state information of the first virtual character while quantifying the environmental perception information. This improves the accuracy of the character action decisions while saving computing resources, making the first virtual character's execution of the role action more realistic.

[0359] Please refer to Figure 19, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device can be implemented as a terminal or server in the above-mentioned embodiment. Specifically, the computer device 1900 includes a central processing unit (CPU) 1901, a system memory 1904 including a random access memory 1902 and a read-only memory 1903, and a system bus 1905 connecting the system memory 1904 and the central processing unit 1901. The computer device 1900 also includes a basic input / output system (I / O system) 1906 that helps transmit information between various components in the computer, and a large-capacity storage device 1907 for storing an operating system 1913, application programs 1914, and other program modules 1915.

[0360] In some embodiments, the basic input / output system 1906 includes a display 1908 for displaying information and an input device 1909, such as a mouse or keyboard, for user input. Both the display 1908 and the input device 1909 are connected to the central processing unit 1901 via an input / output controller 1910 connected to the system bus 1905. The basic input / output system 1906 may also include an input / output controller 1910 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1910 also provides output to a display screen, printer, or other type of output device. The mass storage device 1907 is connected to the central processing unit 1901 via a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 and its associated computer-readable media provide non-volatile storage for the computer device 1900. In other words, the mass storage device 1907 may include computer-readable media (not shown), such as a hard disk or drive.

[0361] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1901, and the one or more programs include instructions for implementing the above-mentioned method, and the central processing unit 1901 executes the one or more programs to implement the method provided by the above-mentioned various method embodiments. According to various embodiments of the present application, the computer device 1900 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1900 can be connected to a network 1912 through a network interface unit 1911 connected to the system bus 1905, or in other words, the network interface unit 1911 can also be used to connect to other types of networks or remote computer systems (not shown). The memory also includes one or more programs, and the one or more programs are stored in the memory, and the one or more programs include the steps performed by the computer device in the method provided by the embodiment of the present application.

[0362] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the action decision method of the virtual character described in any of the above embodiments.

[0363] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the action decision-making method for a virtual character provided in the above aspects.

[0364] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be performed by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium, which can be the computer-readable storage medium included in the memory in the above embodiments, or a separate computer-readable storage medium not incorporated into the terminal. The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the action decision method for the virtual character described in any of the above method embodiments.

[0365] Optionally, the computer-readable storage medium may include: ROM, RAM, solid-state drives (SSDs), or optical disks. Those skilled in the art will appreciate that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or by programs instructing related hardware to accomplish the steps. The programs may be stored in a computer-readable storage medium, which may be a read-only memory, a magnetic disk, or an optical disk.

[0366] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant regions. In addition, before collecting the user's relevant data and during the process of collecting the user's relevant data, this application can display a prompt interface, pop-up window or output voice prompt information. The prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the user's relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window, otherwise the relevant steps of obtaining the user's relevant data are terminated, that is, the user's relevant data is not obtained.

[0367] It should be understood that the "plurality" mentioned in this article refers to two or more. The "first", "second", etc. mentioned in this article are used to distinguish similar objects, and are not used to limit a specific order or sequence. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may also be executed in a non-numbered order, such as two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in an order opposite to that shown in the figure. This embodiment of the application does not limit this.

[0368] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for action decision of a virtual character, executed by a computer device, the method comprising: Acquire character state information of the first virtual character, and acquire environmental perception information of the first virtual character to the current virtual environment, wherein the environmental perception information includes perception information of multiple modes, and the multiple modes respectively represent an information dimension; Fusion of encoding results corresponding to the character state information and the environmental perception information to obtain fusion features, wherein the encoding results are obtained by encoding the character state information and the plurality of perception information respectively by different encoding methods; Performing action decision on the first virtual character based on the fusion feature to obtain a character action; Controlling the first virtual character to perform the character action.

2. The method according to claim 1, wherein: The environmental perception information includes first modality perception information, second modality perception information and third modality perception information; The step of obtaining the environmental perception information of the first virtual character about the current virtual environment includes: Based on the position of the first virtual character in the virtual environment, acquiring the first modal perception information through ray detection, wherein the first modal perception information is a one-dimensional vector and is used to represent the depth of obstacles at the same level around the first virtual character; Based on the character position of the first virtual character in the virtual environment and the orientation of the first virtual character, acquiring the second modal perception information through ray detection, wherein the second modal perception information is a two-dimensional depth map in the orientation direction of the first virtual character, and the second modal perception information is used to characterize the depth of obstacles in the orientation direction of the first virtual character; Based on the character position of the first virtual character in the virtual environment, the third modal perception information of the virtual environment within a preset range of the first virtual character is acquired through ray detection, wherein the third modal perception information is a three-dimensional occupancy grid map, and the third modal perception information is used to characterize the spatial distribution of obstacles within the preset range.

3. The method according to claim 1 or 2, wherein: The acquiring the first modal perception information by ray detection based on the position of the first virtual character in the virtual environment includes: Taking at least two heights of the character position of the first virtual character as starting points, emitting environment perception rays in multiple directions; wherein, when the environment perception rays collide with the surface of an obstacle, they are reflected along the normal direction of the environment perception rays, and the emission directions of different environment perception rays emitted at the same height are located at the same horizontal height; The first modal perception information is generated according to the reflection condition of the environmental perception ray.

4. The method according to any one of claims 1 to 3, wherein: The acquiring the second modality perception information through ray detection based on the character position of the first virtual character in the virtual environment and the orientation of the first virtual character includes: Taking the first virtual character as a starting point, emitting an environment perception ray in the direction of the first virtual character, wherein when the environment perception ray collides with an obstacle surface, the environment perception ray is reflected along the normal direction of the environment perception ray; The second modal perception information is generated according to the reflection condition of the environmental perception ray.

5. The method according to any one of claims 1 to 4, wherein: The acquiring, based on the character position of the first virtual character in the virtual environment, the third modality perception information of the virtual environment within a preset range of the first virtual character through ray detection includes: Dividing the virtual environment within a preset range of the first virtual character using an occupation grid, wherein the occupation grid is composed of a plurality of cubes of the same volume; Environmental perception rays are emitted from starting points in at least two directions in the virtual environment into the occupation grid, and the third modality perception information is generated based on reflection conditions of the environmental perception rays.

6. The method according to any one of claims 1 to 5, wherein: Performing action decision on the first virtual character through an action decision model, wherein the action decision model includes an encoding network and a decision network; The fusion of the encoding results corresponding to the character state information and the environment perception information to obtain the fusion feature includes: Inputting the character state information and the environment perception information into the action decision model, encoding the character state information and the environment perception information through the encoding network, and obtaining a state information encoding result corresponding to the character state information and a perception information encoding result corresponding to the environment perception information; splicing the state information encoding result and the perception information encoding result to obtain a spliced ​​feature code; The splicing feature code is extracted through the coding network feature to obtain the fusion feature.

7. The method according to any one of claims 1 to 6, wherein: The encoding network includes a state information encoder and a perception information encoder, wherein the perception information encoder includes convolution kernels of multiple dimensions, and the convolution kernels of the multiple dimensions are respectively used to perform feature encoding on the perception information of multiple modalities; The step of encoding the character state information and the environment perception information through the encoding network to obtain a state information encoding result corresponding to the character state information and a perception information encoding result corresponding to the environment perception information includes: Encoding the character state information by the state information encoder to obtain the state information encoding result corresponding to the character state information; Encoding the first modal perception information through the one-dimensional convolution kernel in the perception information encoder to obtain a first perception information encoding result; Encoding the second modality perception information through the two-dimensional convolution kernel in the perception information encoder to obtain a second perception information encoding result; The third modality perceptual information is encoded by the three-dimensional convolution kernel in the perceptual information encoder to obtain a third perceptual information encoding result, and the first perceptual information encoding result, the second perceptual information encoding result and the third perceptual information encoding result constitute the perceptual information encoding result.

8. The method according to any one of claims 1 to 7, wherein: The character state information includes absolute state information, and the absolute state information is used to represent state information of the first virtual character that is independent of other virtual characters or obstacles in the virtual environment; After obtaining the character status information of the first virtual character, the method further includes: The absolute state information contained in the character state information is generalized to obtain relative state information, where the relative state information is state information of the first virtual character relative to other virtual characters or obstacles.

9. The method according to any one of claims 1 to 8, wherein: The decision network includes n action output heads, the n action output heads are used to serially output n object sub-actions, different action output heads correspond to different action types, and the n action output heads are serially connected based on the dependency relationship between the action types, the dependency relationship is used to characterize the dependency restriction between sub-actions under different action types, the n object sub-actions constitute the role action, and n is a positive integer greater than 2; The step of making an action decision for the first virtual character based on the fusion feature to obtain a character action includes: Inputting the fused feature into a first action output head in a decision network, and determining a first object sub-action from sub-actions of a first action type through the first action output head in the decision network; The embedded coding vectors corresponding to the determined first object sub-action to the i-1th object sub-action and the fused features are input into the i-th action output head, and the i-th object sub-action is determined from the sub-actions of the i-th action type through the i-th action output head, where i is a positive integer greater than 1.

10. The method according to any one of claims 1 to 9, wherein: Among the n action output heads, at least two of the action output heads correspond to the same action type, the i-th action output head corresponds to a coarse-grained sub-action, and the j-th action output head corresponds to a fine-grained sub-action, where i is less than j and j is a positive integer greater than 2; The determining the i-th object sub-action from the sub-actions of the i-th action type through the i-th action output head includes: Determining the i-th object sub-action from the coarse-grained sub-actions through the i-th action output head in the decision network; The method further comprises: Based on the fusion vector and the embedded coding vector corresponding to the i-th object sub-action, the j-th object sub-action is determined from the fine-grained sub-actions through the j-th action output head, and the action range indicated by the j-th object sub-action is smaller than the action range indicated by the i-th character action.

11. The method according to any one of claims 1 to 10, wherein: Before determining the i-th object sub-action from the sub-actions of the i-th action type through the i-th action output head, the method further includes: Based on the determined first object sub-action to the (i-1)th object sub-action, and the dependency restriction conditions between different sub-actions in different action types, an action mask is performed on the sub-action in the (i)th action type corresponding to the (i)th action output head, wherein at least two sub-actions with a dependency restriction relationship indicated by the dependency restriction conditions cannot be executed at the same time, and the action mask is used to shield unreasonable sub-actions or invalid sub-actions; The determining the i-th object sub-action from the sub-actions of the i-th action type through the i-th action output head includes: The i-th object sub-action is determined from the unmasked sub-actions in the i-th action type through the i-th action output head.

12. The method according to any one of claims 1 to 11, wherein: The method further comprises: Taking the eye position of the first virtual character as a starting point, emitting environment perception rays to various parts of other virtual characters in the direction of the first virtual character; Determining the visibility of various parts of other virtual characters in the virtual environment according to the reflection of the environment perception ray, wherein the visibility is used to indicate whether various parts of other virtual characters are blocked by a cover; The performing action masking on the sub-action in the i-th action type corresponding to the i-th action output head includes: Based on the visual conditions of various parts of other virtual characters, performing action masking on the sub-actions in the character action type, wherein the object sub-actions and the visual conditions of various parts of other virtual characters have the dependency restriction relationship; The determining the i-th object sub-action from the sub-actions of the i-th action type through the i-th action output head includes: The object sub-action is determined from the unmasked sub-actions in the character action type through the character action output head.

13. The method according to any one of claims 1 to 12, wherein: Among the n action output heads, there is a first steering action output head corresponding to a coarse-grained steering action, and there is a second steering action output head corresponding to a fine-grained steering action, the first steering action output head is used to determine a first steering angle, and the second steering action output head is used to determine a second steering angle; In the case where the first virtual character has the function of aiming at other virtual characters, the method further comprises: Determining an ideal turning angle based on the orientation of the first virtual character, the position coordinates of the first virtual character and the position coordinates of other virtual characters, and determining an aiming distance between the first virtual character and the other virtual characters; Determining an effective aiming range based on the aiming distance and the width of other virtual characters; The performing action masking on the sub-action in the i-th action type corresponding to the i-th action output head includes: Determining an angle difference between the first steering angle and the ideal steering angle based on a first steering angle determined by the first steering action output head; Determining an effective fine-grained steering angle based on the angle difference and the effective aiming range; Performing action masking on the steering angles in the coarse-grained steering action that do not belong to the valid fine-grained steering angles; The determining the i-th object sub-action from the sub-actions of the i-th action type through the i-th action output head includes: The second steering action output head is used to determine the second steering angle from the effective fine-grained steering angles that are not masked in the fine-grained steering action.

14. The method according to any one of claims 1 to 13, wherein: Before obtaining the character status information of the first virtual character, the method includes: Acquire sample character state information of the first virtual character and sample environment perception information of the first virtual character about the current virtual environment, wherein the sample environment perception information includes sample perception information of multiple modalities, and the multiple modalities respectively represent an information dimension; The sample encoding results corresponding to the sample role state information and the sample environment perception information are fused through the encoding network of the action decision model to obtain the sample fusion feature, wherein the encoding network encodes the sample role state information and the sample environment perception information in different encoding modes. Characterizing the character status information and the sample environment perception information of different modalities; Based on the sample fusion features, the action decision model is trained by reinforcement learning.

15. The method according to any one of claims 1 to 14, wherein: The step of training the action decision model by reinforcement learning based on the sample fusion features includes: Inputting the sample fusion features into the decision network in the action decision model to obtain an estimated character action; controlling the first virtual character to execute the estimated character action, and obtaining an estimated action execution result of the first virtual character; determining an estimated action execution reward based on the estimated action execution result of the first virtual character; Model parameters of the action decision model are updated based on the estimated action execution reward.

16. The method according to any one of claims 1 to 15, wherein: The estimated action execution reward includes at least one of a reasonable action reward, a cover utilization reward, a visibility reward, and a movement dispersion reward; The reasonable action reward refers to a reward that complies with the operation logic of the real player when the first virtual character performs the estimated character action; The cover utilization reward refers to a reward for the first virtual character being located adjacent to a cover when the first virtual character performs the estimated character action; The visibility reward refers to a reward when the virtual character is in a valid attack position for other virtual characters when performing the estimated character action; The movement dispersion reward refers to a reward for a plurality of consecutive movements with a reduced density when the first virtual character executes the estimated character action.

17. The method according to any one of claims 1 to 16, wherein: The estimated action execution result includes the position point of the first virtual character after performing the first n+1 actions; The determining of the estimated action execution reward based on the estimated action execution result of the first virtual character includes: Based on the position point of the first virtual character after performing the estimated character action for the n+1th time, determining a first closeness between the position point after performing the n+1th action and the position point after performing the action for the previous n times, and determining a second closeness between the position point after performing the nth action and the position point after performing the action for the previous n-1 times; When the first compactness is less than the second compactness, the movement dispersion reward is determined as the estimated action execution reward.

18. The method according to any one of claims 1 to 17, wherein: The estimated action execution result includes the sample environment perception information after the first virtual character executes the estimated character action, and the sample environment perception information includes a two-dimensional depth map, and the two-dimensional depth map is used to represent the obstruction of the first virtual character by obstacles in the direction in which the first virtual character is facing; The determining of the estimated action execution reward based on the estimated action execution result of the first virtual character includes: Dividing the two-dimensional depth map into regions to obtain at least two depth regions; Determine a first masking rate of a first depth region and second masking rates of at least two second depth regions adjacent to the first depth region, wherein the first depth region is located at a center of a field of view of the first virtual character; Determining the bunker utilization reward based on the minimum difference between the first concealment rate and the second concealment rate, wherein the degree of the bunker utilization reward is positively correlated with the minimum difference; The cover utilization reward is determined as the estimated action performance reward.

19. The method according to any one of claims 1 to 18, wherein: The estimated action execution result includes a mutual visual relationship between the first virtual character and other virtual characters, wherein the mutual visual relationship is used to represent the visual conditions of the first virtual character and other virtual characters relative to each other; The determining of the estimated action execution reward based on the estimated action execution result of the first virtual character includes: When the mutual visibility relationship indicates that the first virtual character is not within the visible range of other virtual characters, and other virtual characters are within the visible range of the first virtual character, determining that the estimated action execution reward is the positive visibility reward; When the mutual visibility relationship indicates that the first virtual character is within the visibility range of other virtual characters, and other virtual characters are not within the visibility range of the first virtual character, the estimated action execution reward is determined to be the reverse visibility reward.

20. The method according to any one of claims 1 to 19, wherein: The method further comprises: Determining an ideal action of the first virtual character based on the sample character state information and the sample environment perception information of the first virtual character; The determining of the estimated action execution reward based on the estimated action execution result of the first virtual character includes: When the estimated character action is consistent with the ideal action, the reasonable action reward is determined as the estimated action execution reward.

21. The method according to any one of claims 1 to 20, wherein: There is a turning sub-action in the character action; The step of determining an ideal action of the first virtual character based on the sample character state information and the sample environment perception information of the first virtual character includes: Taking the first virtual character as a starting point, emitting environment perception rays in multiple directions, different environment perception rays are located at the same horizontal height, and when the environment perception rays collide with the surface of an obstacle, they are reflected along the normal direction of the environment perception rays; When the reflection condition of the environment perception ray indicates that the projection distance of the first environment perception ray is less than a distance threshold, determining the emission direction of the environment perception ray adjacent to the first environment perception ray and having a projection distance greater than the distance threshold as the ideal turning direction, where the projection distance refers to the distance from the starting point to the reflection point of the environment perception ray; When the estimated character action is consistent with the ideal action, determining the reasonable action reward as the estimated action execution reward includes: In a case where the steering direction indicated by the steering sub-action is consistent with the ideal steering direction, the reasonable action reward is determined as the estimated action execution reward.

22. The method according to any one of claims 1 to 21, wherein: There is at least one other virtual character in the game in which the first virtual character is playing, and the other virtual character executes the action output by the second action decision model; The method further comprises: When a first action decision model corresponding to the first virtual character completes a round of training, the second action decision model is trained based on the trained first action decision model, and the second action decision model after the training is completed is stored in the opponent model pool; Selecting the trained second action decision model from the opponent model pool, and constructing a new game, wherein the new game includes the first virtual character executing the estimated character action output by the first action decision model, and other virtual characters executing the action output by the second action decision model; The first action decision model is trained in the new game.

23. The method according to any one of claims 1 to 22, wherein: The method further comprises: Upon completion of the i-th round of training of the action decision model, obtaining game data of the i-th round of training games; Determine a game index and an anthropomorphic index based on the game data of at least two rounds of training games, wherein the anthropomorphic index refers to an actual execution ratio of a specific action in the game data of at least the training games; When the game indicator reaches the training completion standard and the anthropomorphic indicator matches the action execution ratio of the specific action performed by the real player in the game, it is determined that the action decision model has completed training.

24. A virtual character action decision device, the device comprising: An acquisition module, used to acquire character state information of the first virtual character, and to acquire environmental perception information of the first virtual character to the current virtual environment, wherein the environmental perception information includes perception information of multiple modes, and the multiple modes respectively represent an information dimension; The encoding module is used to fuse the encoding results corresponding to the character state information and the environmental perception information to obtain a fusion feature, wherein the encoding result is obtained by encoding the character state information and multiple perception information in different ways. The results obtained by feature encoding respectively; A decision module, configured to make an action decision for the first virtual character based on the fusion feature to obtain a character action; A control module is used to control the first virtual character to perform the character action.

25. A computer device, comprising a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the action decision method of a virtual character as described in any one of claims 1 to 23.

26. A computer-readable storage medium, wherein at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement the action decision method of a virtual character as described in any one of claims 1 to 23.

27. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the action decision method for a virtual character as described in any one of claims 1 to 23.

Images