Methods, devices, terminal equipment and storage media for controlling virtual characters

Abstract

This invention provides a method, apparatus, terminal device, and storage medium for controlling a virtual character, comprising: acquiring target voxel data of a three-dimensional virtual scene in which the virtual character is located, and state parameters of the virtual character in the three-dimensional virtual scene; determining an action to be performed based on the target voxel data and the state parameters using a target pathfinding model; wherein the target pathfinding model is trained using a reinforcement learning algorithm; and controlling the virtual character to move within the three-dimensional virtual scene based on the action to be performed. This invention can significantly improve the pathfinding success rate and also enhance the anthropomorphism of the virtual character.

Description

Technical Field

[0001] This invention relates to the field of game technology, and in particular to a method, apparatus, terminal device, and storage medium for controlling virtual characters. Background Technology

[0002] In complex 3D game environments, NPCs (non-player characters) enhance the player experience in various ways and are an important component of the game. NPCs' pathfinding abilities allow them to move within the game map. Furthermore, to mimic player movement, NPCs need to navigate around obstacles and utilize climbing, jumping, and movement skills to increase their speed, giving them anthropomorphic pathfinding strategies. However, existing games suffer from problems such as low NPC pathfinding success rates and significant differences in behavior patterns compared to player characters. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, device, terminal equipment and storage medium for controlling virtual characters, which can significantly improve the success rate of pathfinding and also improve the anthropomorphism of virtual characters.

[0004] In a first aspect, embodiments of the present invention provide a method for controlling a virtual character, comprising: acquiring target voxel data of a three-dimensional virtual scene in which the virtual character is located, and state parameters of the virtual character in the three-dimensional virtual scene; determining an action to be performed based on the target voxel data and the state parameters using a target pathfinding model; wherein the target training model is trained using a reinforcement learning algorithm; and controlling the virtual character to move in the three-dimensional virtual scene based on the action to be performed.

[0005] In one implementation, the state parameters include current position parameters and target position parameters; the step of determining the action to be executed based on the target voxel data and the state parameters using a target pathfinding model includes: acquiring a depth image of the three-dimensional virtual scene and historical actions corresponding to the previous position parameter of the current position parameter; and determining the action to be executed corresponding to the current position parameter based on the target voxel data, the state parameters, the depth image, and the historical actions using a target pathfinding model.

[0006] In one embodiment, the target pathfinding model includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to one of the processing sub-networks. The step of determining the action to be executed corresponding to the current position parameter based on the target voxel data, the state parameters, the depth image, and the historical actions by the target pathfinding model includes: using the target voxel data, the state parameters, the depth image, and the historical actions as input data for each input terminal; extracting feature information from the input data of each input terminal through each processing sub-network, and performing feature concatenation processing on each feature information to obtain concatenated feature information; determining the output data of each output terminal based on the concatenated feature information by the multilayer perceptron, and using the output data of each output terminal as the action to be executed.

[0007] In one embodiment, the input terminals include a voxel input terminal, a state input terminal, a depth input terminal, and an action input terminal. The processing sub-network includes a voxel processing sub-network, a fully connected sub-network, a residual sub-network, and a Transformer sub-network. The voxel input terminal is connected to the voxel processing sub-network, the state input terminal is connected to the fully connected sub-network, the depth input terminal is connected to the residual sub-network, and the action input terminal is connected to the Transformer sub-network. The step of extracting feature information of the input data of each input terminal through each processing sub-network includes: extracting feature information of the target voxel data through the voxel processing sub-network; extracting feature information of the state parameters through the fully connected sub-network; extracting feature information of the depth image through the residual sub-network; and extracting feature information of the historical action through the Transformer sub-network.

[0008] In one embodiment, the method further includes: acquiring multiple first scene sets and second scene sets; wherein the scene complexity of the second scene set is greater than the scene complexity of the first scene set; using the reinforcement learning algorithm, performing a first-stage training on an initial pathfinding model based on the first scene set until the initial pathfinding model meets a first preset requirement; using the reinforcement learning algorithm, performing a second-stage training on the initial pathfinding model that meets the first preset requirement based on the second scene set until the initial pathfinding model meets a second preset requirement to obtain a target pathfinding model.

[0009] In one implementation, the step of using the reinforcement learning algorithm to perform a first-stage training of the initial pathfinding model based on the first scene set includes: using the first voxel data of the first preset scene in the first scene set as the model state of the initial pathfinding model, and using the first action of the virtual character in the first preset scene as the model action of the initial pathfinding model; determining the reward function corresponding to the pathfinding model, and using the reinforcement learning algorithm to perform a first-stage training of the initial pathfinding model based on the model state, the model action, and the reward function.

[0010] In one implementation, the step of determining the reward function corresponding to the initial pathfinding model includes: determining an intermediate reward based on the current point of the virtual character in the first preset scene, the previous point corresponding to the current point, and the target point; or determining a time penalty based on the movement time taken by the virtual character to move from the previous point to the current point; or determining a sparse reward based on the initial point of the virtual character in the first preset scene and the target point; and determining one or more of the intermediate reward, the time penalty, and the sparse reward as the reward function corresponding to the initial pathfinding model.

[0011] In one embodiment, the action to be executed includes character movement parameters, which include character movement direction and character behavior; the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be executed includes: triggering the virtual character to execute the character behavior and moving in the three-dimensional virtual scene according to the character movement direction.

[0012] In one embodiment, the action to be performed further includes camera movement parameters, which include a horizontal camera movement direction and a vertical camera movement direction; after the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be performed, the method further includes: controlling the virtual camera of the three-dimensional virtual scene to move in the three-dimensional virtual scene according to the horizontal camera movement direction and the vertical camera movement direction.

[0013] In one embodiment, the step of obtaining target voxel data of the three-dimensional virtual scene in which the virtual character is located includes: obtaining initial voxel data of the three-dimensional virtual scene in which the virtual character is located; assigning values ​​to each element in the initial voxel data according to the material of the objects contained in the three-dimensional virtual scene to obtain target voxel data; wherein, the elements in the target voxel data are used to characterize the material of the object corresponding to the element; or, associating the initial voxel data with the material of the objects contained in the three-dimensional virtual scene respectively, and for each material, assigning values ​​to the elements in the initial voxel data associated with the material that match the material to obtain the target voxel data associated with the material.

[0014] Secondly, embodiments of the present invention also provide a control device for a virtual character, comprising: an acquisition module, configured to acquire target voxel data of a three-dimensional virtual scene in which the virtual character is located, and state parameters of the virtual character in the three-dimensional virtual scene; an action determination module, configured to determine an action to be performed based on the target voxel data and the state parameters using a target pathfinding model; wherein the target training model is trained using a reinforcement learning algorithm; and a control module, configured to control the virtual character to move in the three-dimensional virtual scene based on the action to be performed.

[0015] Thirdly, embodiments of the present invention also provide a terminal device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method described in any of the first aspects.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method described in any of the first aspects.

[0017] This invention provides a method, apparatus, terminal device, and storage medium for controlling a virtual character. First, it acquires target voxel data of the 3D virtual scene in which the virtual character resides, as well as the virtual character's state parameters within the 3D virtual scene. Then, it determines the action to be executed based on the target voxel data and state parameters using a target pathfinding model. Finally, based on the action to be executed, it controls the virtual character's movement within the 3D virtual scene. This method determines the action to be executed based on the target voxel data and state parameters of the 3D virtual scene using a target pathfinding model. The target voxel data allows the virtual character to have a more refined perception of the 3D virtual scene, enabling control of its movement within the scene, thereby significantly improving the pathfinding success rate and enhancing the anthropomorphism of the virtual character.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a virtual character control method provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a three-dimensional virtual game scene and its initial voxel data provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a target pathfinding model provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of a voxel processing subnetwork provided in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of another three-dimensional virtual game scene and its initial voxel data provided in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of a control device for a virtual character provided in an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Currently, NPCs can also serve as AI (Artificial Intelligence) companions in games, and the intelligence level of these AI companions directly affects a game's entertainment value and lifespan. Traditional game AI companions primarily employ rule-based methods, with typical examples including Finite State Machines (FSMs) and Behavior Trees (BRTs). These rule-based methods are still widely used in various commercial games today. Traditional pathfinding algorithms like A-Star are mostly suitable for pathfinding in a 2D-like plane. However, due to the significant vertical differences in 3D scenes and unique actions such as flight and teleportation, including climbing and flying motion patterns, the success rate of traditional pathfinding algorithms is greatly reduced. Furthermore, the robot's behavior patterns will differ significantly from those of other human players, making it easier for players to spot.

[0030] Specifically, the existing 3D scene pathfinding schemes are mainly of the following two types: (1) using graph structures to call pathfinding algorithms such as A-star, and (2) using behavior trees for pathfinding.

[0031] For the above scheme (1), the formal expression of the graph is G = (V, E), where V is the set of vertices, and E and V are a binary relation, which can be understood as edges. For example, if there is an edge that runs from vertex U to vertex V, then E can be represented by (u, v). The distinction between directed and undirected graphs is also based on whether the edges have a direction.

[0032] The existing technical solution first models the game map into the graph structure described above, and then calls pathfinding algorithms such as A-Star to find the optimal route for the modeled graph. In the A-Star algorithm, if g(n) represents the actual distance from the starting point to any vertex n, and h(n) represents the estimated distance from any vertex n to the target vertex (which varies depending on the evaluation function used), then the estimation function of the A-Star algorithm is: f(n) = g(n) + h(n). This formula follows the following characteristics: (1) If g(n) is 0, that is, only the evaluation function h(n) from any vertex n to the target is calculated, without calculating the distance from the starting point to vertex n, the algorithm is transformed into a best-first search using a greedy strategy, which is the fastest, but may not find the optimal solution; (2) If h(n) is not greater than the actual distance from vertex n to the target vertex, the optimal solution can definitely be found, and the smaller h(n) is, the more nodes need to be calculated, and the lower the efficiency of the algorithm. Common evaluation functions include Euclidean distance, Manhattan distance, and Chebyshev distance; (3) If h(n) is 0, that is, only the shortest path g(n) from the starting point to any vertex n is needed, without calculating any evaluation function h(n), it is transformed into a shortest path problem, that is, Dijkstra's algorithm, which requires the most vertices to be calculated.

[0033] In practical applications, we usually take h(n) as the Manhattan distance between two nodes. In this way, the evaluation function f(n) will be more or less constrained by the distance estimate h(n). The closer the node is to the target point, the smaller the h value, and the relatively smaller the f value. This ensures that the shortest path search proceeds in the direction of the destination, which is significantly better than Dijkstra's algorithm, which searches in all directions without any direction.

[0034] The process of scheme (1) includes: generating an empty open table and a close table; putting the starting point S into the open table; ending if the open table is empty, and if the open table is not empty, sorting the nodes in the open table according to the evaluation function to obtain the node U with the smallest evaluation function value; removing U from the open table and putting U into the close table; if U is the target node, backtracking the close table to obtain the path and ending, and if U is not the target node, adding all of U's adjacent nodes to the open table according to the adjacent node rules.

[0035] However, the above scheme (1) requires converting the game scene into a graph structure, which means dividing the scene into several polygons. This will cause the AI ​​companion to only walk on the edges of these polygons when navigating, and will not reach the interior of the polygons, resulting in a very simple route. In addition, in large and complex 3D scenes, the divided graph structure may not be connected, and pathfinding algorithms such as A-star can only perform pathfinding in connected graphs, and cannot be used in multiple disconnected graphs.

[0036] Regarding the above scheme (2), rules are manually written to enable the agent to exhibit certain human-like behaviors in specific scenarios, such as jumping when there is a height difference between the target point and the current position, and sprinting otherwise. However, if the action system in the game is complex, not only is it difficult to write perfect rules, but the AI ​​partner may also be unable to use the fast movement actions in the game to find the truly fastest path.

[0037] Based on this, the present invention provides a method, device, terminal equipment and storage medium for controlling virtual characters, which can significantly improve the success rate of pathfinding and also improve the anthropomorphism of virtual characters.

[0038] In one embodiment of this disclosure, the virtual character control method can run on a local terminal device or a terminal device. When the virtual character control method runs on a terminal device, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a terminal device and a client device.

[0039] In an optional implementation, various cloud applications, such as cloud gaming, can run under the cloud interaction system. Taking cloud gaming as an example, cloud gaming refers to a gaming method based on cloud computing. In the cloud gaming operating mode, the game program and the game screen presentation are separate. The storage and execution of information display methods in the game are completed on the cloud gaming terminal device. The client device is used for data reception, transmission, and game screen presentation. For example, the client device can be a display device with data transmission capabilities located close to the user, such as a mobile terminal, television, computer, or PDA; however, information processing is performed by the cloud gaming terminal device in the cloud. When playing the game, the player operates the client device to send operation commands to the cloud gaming terminal device. The cloud gaming terminal device runs the game according to the operation commands, encodes and compresses game screen data, returns it to the client device via the network, and finally, the client device decodes and outputs the game screen.

[0040] In an optional implementation, taking a game as an example, the local terminal device stores the game program and is used to display the game screen. The local terminal device is used to interact with the player through a graphical user interface (GUI), i.e., conventionally by downloading, installing, and running the game program via an electronic device. The local terminal device can provide the GUI to the player in various ways, such as rendering it on the terminal's display screen or providing it to the player via holographic projection. For example, the local terminal device can include a display screen for displaying the GUI, which includes game screens, and a processor for running the game, generating the GUI, and controlling the display of the GUI on the display screen.

[0041] In one possible implementation, this disclosure provides a method for controlling a virtual character, which provides a graphical user interface through a terminal device. The terminal device can be either the aforementioned local terminal device or a client device in the aforementioned cloud interaction system.

[0042] The technical solutions of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0043] To facilitate understanding of this embodiment, a method for controlling a virtual character disclosed in this invention will first be described in detail. (See also...) Figure 1 The diagram shows a flowchart of a method for controlling a virtual character. The method mainly includes the following steps S102 to S106:

[0044] Step S102: Obtain the target voxel data of the 3D virtual scene where the virtual character is located, and the state parameters of the virtual character in the 3D virtual scene. The virtual character can be an AI companion (i.e., an NPC) or a game character controlled by the player. The target voxel data is a 3D matrix or a 4D matrix, where each element represents the material of an object at a certain coordinate in the 3D virtual scene. The state parameters can include current position parameters and target position parameters. In one implementation, since the game itself contains voxel information, the initial voxel data of the 3D virtual scene can be directly obtained. Material processing of the initial voxel data yields the target voxel data representing the material of objects at each coordinate in the 3D virtual scene. Furthermore, the target position parameters can be pre-configured, or the coordinates of the virtual character in the 3D virtual scene can be directly read as the current position parameters.

[0045] Step S104: The target pathfinding model determines the action to be executed based on target voxel data and state parameters. The action to be executed can include various types of actions, such as character-related actions (also referred to as character movement parameters) and / or camera-related actions (also referred to as camera movement parameters). Character-related actions can include things like character movement direction and character behavior, while camera-related actions can include things like camera horizontal movement direction and camera vertical movement direction. Furthermore, the target pathfinding model can be trained using reinforcement learning. In one embodiment, the input to the target pathfinding model includes target voxel data and state parameters, as well as a depth image of the 3D virtual scene and historical actions corresponding to the previous position parameters of the current position parameter; the output is the action to be executed.

[0046] Step S106: Based on the action to be executed, control the virtual character to move within the 3D virtual scene. In one embodiment, the virtual character's movement within the 3D virtual scene can be controlled according to a character-type action in the action to be executed. In another embodiment, the virtual camera can also be controlled to move within the 3D virtual scene according to a camera-type action in the action to be executed, so as to capture game footage of the virtual character moving within the 3D virtual scene using the virtual camera.

[0047] The virtual character control method provided in this invention can determine the action to be executed based on the target voxel data and state parameters of the three-dimensional virtual scene through the target pathfinding model. The target voxel data enables the virtual character to have a more refined perception of the three-dimensional virtual scene, and on this basis, controls the virtual character to move in the three-dimensional virtual scene, thereby significantly improving the pathfinding success rate and also improving the anthropomorphism of the virtual character.

[0048] To facilitate understanding of step S102 above, this embodiment of the invention provides an implementation method for obtaining target voxel data of a three-dimensional virtual scene in which a virtual character is located, as shown in steps (1) to (2) below:

[0049] (1) Obtain the initial voxel data of the 3D virtual scene in which the virtual character is located. In practical applications, voxel data can finely depict the surrounding environment, such as... Figure 2 The diagram illustrates a 3D virtual game scene and its initial voxel data. Different colors in the initial voxel data represent different object materials, such as grass, rocks, tiles, and wood. Voxel data is a commonly used 3D data table structure, using fixed-size cubes as the smallest unit; it represents pixels in 3D. Due to its structured nature, voxel data can be directly used as input for target pathfinding models. Furthermore, the voxel data in the game removes invalid information such as weeds that obstruct the view. Therefore, compared to existing technologies that directly use scene images or depth images of 3D virtual scenes, this invention can more efficiently acquire actions to be performed.

[0050] (2) Determine the target voxel data based on the object materials and initial voxel data contained in the 3D virtual scene. In practical applications, the material representation and accuracy of voxel data are important factors affecting accuracy. Therefore, to more intuitively reflect the material representation of voxel data, it is necessary to perform corresponding material processing on the initial voxel data. For example, taking a certain 3D virtual scene as an example, there are 21 types of object materials in this game. Assume that before being used as input data for the target pathfinding model, the initial voxel data is a 3D matrix: m*n*p, and perform material processing on this 3D matrix. For details, please refer to the following methods one to two:

[0051] Method 1: Based on the materials of the objects contained in the 3D virtual scene, assign values ​​to each element in the initial voxel data to obtain the target voxel data. The elements in the target voxel data represent the material of the corresponding object, with each assigned value corresponding to a specific object material.

[0052] In one implementation, 21 object materials are represented by values ​​from 0 to 20. For example, in the three-dimensional matrix, one element corresponds to a value of "1", which represents that the object material is wood, and another element corresponds to a value of "2", which represents that the object material is stone. By assigning a value to each element in the three-dimensional matrix, the object material represented by each element can be determined. The three-dimensional matrix after assignment is the target voxel data. It should be noted that the number of object materials and the values ​​used can be set according to actual needs, and this application does not impose specific limitations.

[0053] In another implementation, 21 object materials are represented by values ​​from 0 to 20, and these values ​​are normalized by dividing each value by 20 to ensure that the values ​​corresponding to all object materials are between 0 and 1. The normalized values ​​are then assigned to each element in the three-dimensional matrix to obtain the target voxel data. By normalizing the values, the computational difficulty of the subsequent target pathfinding model can be significantly reduced, thereby improving pathfinding efficiency to some extent.

[0054] Method 2 involves associating the initial voxel data with the materials of objects within the 3D virtual scene. For each material, the element in the initial voxel data associated with that material that matches that material is assigned a value, thus obtaining the target voxel data associated with that material. For example, assuming there are 21 types of object materials in the game, there are 21 fourth-dimensional channels, and each fourth-dimensional channel is associated with a 3D matrix, forming a 4-dimensional matrix: m*n*p*21. For each object material's associated 3D matrix, the element corresponding to that object material can be set to 1, and the remaining elements can be set to 0, thereby obtaining the target voxel data associated with each object material.

[0055] To facilitate understanding of the aforementioned step S104, this embodiment of the invention also provides an implementation method for determining the action to be executed based on target voxel data and state parameters using a target pathfinding model. First, a depth image of the 3D virtual scene and the historical actions corresponding to the previous position parameter of the current position parameter are acquired. Then, the action to be executed corresponding to the current position parameter is determined using the target pathfinding model based on the target voxel data, state parameters, depth image, and historical actions. The depth image is used to represent the depth information of the 3D virtual scene, and the historical actions are the actions to be executed output by the target pathfinding model based on the previous position parameter and target voxel data.

[0056] Furthermore, this embodiment of the invention provides a target pathfinding model, which includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to a processing sub-network. Based on this, this embodiment of the invention provides an implementation method for determining the action to be executed corresponding to the current position parameters, as shown in steps 1 to 3 below:

[0057] Step 1: Use the target voxel data, state parameters, depth image, and historical actions as input data for each input device. See [link / reference] Figure 3 The diagram shown is a schematic of a target pathfinding model. Figure 3 The target pathfinding model is illustrated with input terminals including voxel input, state input, depth input, and action input. The processing subnetworks include a voxel processing subnetwork (3dconv), a fully connected subnetwork (FC), a residual subnetwork, and a Transformer subnetwork. The voxel input is connected to the voxel processing subnetwork, the state input is connected to the fully connected subnetwork, the depth input is connected to the residual subnetwork (ResNet), and the action input is connected to the Transformer subnetwork. In practical implementation, target voxel data can be input through the voxel input, current and target position parameters can be input through the state input, depth images can be input through the depth input, and historical actions can be input through the action input.

[0058] Step 2 involves extracting feature information from the input data at each input end through each processing sub-network, and then concatenating these feature information pieces to obtain concatenated feature information. (As mentioned above...) Figure 3 Based on this, embodiments of the present invention provide an implementation method for extracting feature information of input data from each input terminal. Feature information of target voxel data can be extracted through a voxel processing sub-network; feature information of state parameters can be extracted through a fully connected sub-network; feature information of depth images can be extracted through a residual sub-network; and feature information of historical actions can be extracted through a Transformer sub-network.

[0059] In practical applications, for the aforementioned target voxel data, a voxel processing subnetwork can be used to flatten it into a vector of fixed length. See [link to relevant documentation]. Figure 4 The diagram shown is a structural schematic of a voxel processing subnetwork. Figure 4Several optional voxel processing subnetworks are illustrated. For example, an MLP (Multilayer Perceptron) can be used to flatten the target voxel data into a vector form, outputting a fixed-length vector; alternatively, a 3D-CNN (Convolutional Neural Network) can be used to flatten the target voxel data output into a fixed-length vector; or, a 2D-CNN can be used to flatten the target voxel data output into a fixed-length vector. In practical applications, the 3D-CNN and 2D-CNN structures mentioned above can be implemented using the ResNet architecture.

[0060] For the current position parameters and target position parameters mentioned above, a fully connected layer can be used for processing; for the historical actions mentioned above, a Transformer sub-network can be used to extract the correlation features between historical actions; for the depth image mentioned above, a residual sub-network can be used to extract features from the depth image.

[0061] Step 3: Based on the spliced ​​feature information, the multilayer perceptron determines the output data of each output terminal, and uses the output data of each output terminal as the action to be executed. Please continue to the next step. Figure 3 , Figure 3 The diagram illustrates that a multilayer perceptron can be implemented using three fully connected layers. The multilayer perceptron connects to four outputs, outputting four actions: character movement direction (move), character behavior (other), camera horizontal movement direction (yaw), and camera vertical movement direction (pitch). These four actions constitute the agent's entire action space. In practice, it should be noted that these four actions are discrete. Character movement direction includes "no-op" (idle) and moving in eight directions; camera horizontal movement direction includes moving the virtual camera left or right; camera vertical movement direction includes moving the virtual camera up or down; and character behavior includes sprinting, jumping, sliding, and using items for rapid movement.

[0062] Regarding the target pathfinding model provided in the foregoing embodiments, this embodiment of the invention also provides an implementation method for training the target pathfinding model using a reinforcement learning algorithm, as detailed in steps a to c below:

[0063] Step a: Obtain multiple first scene sets and second scene sets. The scene complexity of the second scene set is greater than that of the first scene set. For example, the first scene set may include a specified number of typical terrain areas, such as temple areas, swamp areas, mine areas, and grassland areas, where the density and shape of obstacles differ. The second scene set may include the entire game map.

[0064] Step b: Using a reinforcement learning algorithm, the initial pathfinding model is trained in the first stage based on the first scene set until the initial pathfinding model meets the first preset requirement. The first preset requirement can be the arrival rate of the virtual object reaching the target point. When the probability of the virtual object reaching the target point while being controlled by the initial pathfinding model in the first scene set is greater than the aforementioned arrival rate, it can be determined that the initial pathfinding model meets the first preset requirement.

[0065] Step c: Using a reinforcement learning algorithm, the initial pathfinding model that meets the first preset requirement is trained in the second stage based on the second scene set until the initial pathfinding model meets the second preset requirement to obtain the target pathfinding model. The second preset requirement can also be the arrival rate of the virtual object reaching the target point. When the probability of the virtual object reaching the target point using the initial pathfinding model in the second scene set is greater than the aforementioned arrival rate, it can be determined that the initial pathfinding model meets the second preset requirement, and the initial pathfinding model that meets the second preset requirement is used as the target pathfinding model.

[0066] In practical applications, the training schemes for the first and second phases of training mentioned above are different, with only minor differences. For example, in the second phase of training, the maximum step size of the task needs to be increased because the terrain of the entire map is more complex.

[0067] Taking the first training phase as an example, an implementation method for training the initial pathfinding model is provided, as shown in steps b1 to b2 below:

[0068] Step b1 involves using the first voxel data of the first preset scene in the first scene set as the model state of the initial pathfinding model, and using the first action of the virtual character in the first preset scene as the model action of the initial pathfinding model. In one embodiment, this invention employs reinforcement learning to train the initial pathfinding model, wherein the state s of the initial pathfinding model is the aforementioned first voxel data, which is also the target voxel data during the training phase, and the state a of the initial pathfinding model is the first action, which is also the action to be executed during the training phase.

[0069] In practical applications, the accuracy of voxel data also affects environmental characterization, such as... Figure 5The diagram illustrates another 3D virtual game scene and its initial voxel data. For the same size m*n*p 3D matrix, the terrain range it can represent varies greatly depending on the precision. Higher precision voxel data represents a smaller map range with more detail for the same matrix size; lower precision voxels improve the represented map range by sacrificing local detail. The choice of voxel precision during training needs to comprehensively consider the required field of view and precision for the actual problem, as well as the processing speed of the neural network. For a pathfinding AI companion, preferably, 1m and 0.5m precision can be used, which is sufficient for the AI ​​companion to perceive the environment and have a surrounding field of view.

[0070] Step b2: Determine the reward function corresponding to the initial pathfinding model, and use reinforcement learning algorithm to perform the first stage training of the initial pathfinding model based on the model state, model action, and reward function. This embodiment of the invention provides an implementation method for determining the reward function r, specifically: (1) Determine the intermediate reward based on the current point of the virtual character in the first preset scene, the previous point corresponding to the current point, and the target point; (2) Determine the time penalty based on the movement time taken by the virtual character from the previous point to the current point; (3) Determine the sparse reward based on the initial point and target point of the virtual character in the first preset scene; (4) Determine one or more of the intermediate reward, time penalty, and sparse reward as the reward function corresponding to the initial pathfinding model. The reward function r provided in this embodiment of the invention adopts a combination of sparse reward and intermediate reward. The sparse reward is the final reward given when the agent reaches the target point and ends the current round. To avoid excessive learning difficulty due to overly sparse rewards, the intermediate reward adopts an immediate distance reward to encourage the AI ​​partner to move closer to the target point. The calculation method is the distance from the current point to the target point minus the distance from the previous point to the target point. To encourage the AI ​​partner to reach the target point faster and learn to use fast-movement items for long-distance rapid movement, time-related penalties are added to the intermediate rewards, with a small penalty for each decision made. The deep reinforcement learning algorithm used in this embodiment of the invention enables the intelligent agent to continuously explore and interact with the game environment. By maximizing the reward function r, it autonomously learns various combinations of actions in pathfinding, as well as avoiding obstacles and leaving enclosed areas such as houses.

[0071] In practical applications, the training process of the second training phase can be referred to the first training phase described above, and will not be repeated here in this embodiment of the invention. In complex 3D virtual environments, it is often very difficult for an AI companion to learn all the pathfinding skills at once. Therefore, this embodiment of the invention introduces a course-based training method, allowing the AI ​​companion to learn various skills from easy to difficult. To enable the AI ​​companion to learn to use special props for rapid movement, this embodiment of the invention adopts a method of giving rewards for using the prop in the early stages to encourage the AI ​​companion to try the action. However, after its usage rate reaches a certain level, the reward standard is increased; that is, only high-quality use of the prop will result in a reward, gradually improving its mastery of the action.

[0072] In one implementation, when controlling a virtual character to move within a three-dimensional virtual scene, the virtual character can be triggered to perform character actions and move within the three-dimensional virtual scene according to the character's movement direction. For example, the virtual character can be triggered to use an item to quickly move in the character's movement direction.

[0073] In another implementation, the virtual camera of the three-dimensional virtual scene can be controlled to move in the three-dimensional virtual scene according to the horizontal and vertical movement directions of the camera, thereby capturing images of the virtual character moving in the three-dimensional virtual scene through the virtual camera.

[0074] The virtual character control method provided in this embodiment of the invention, by applying a deep reinforcement learning algorithm, comprehensively uses scalar information such as position coordinates, and uses voxel information in the game for modeling, enables the pathfinding AI partner to autonomously explore and learn in complex three-dimensional scenes. Compared with traditional pathfinding algorithms, this embodiment of the invention has a higher pathfinding success rate and a higher degree of anthropomorphism. Specifically, (1) using voxel information in the game for modeling, the pathfinding AI partner has a fine perception of the surrounding three-dimensional virtual environment and can obtain information such as terrain and material; (2) using a deep neural network to process state perception: including using 3D-CNN and ResNet to process the voxel three-dimensional matrix, using the Transformer structure to process the historical information two-dimensional matrix, and using FC to process other scalar information; (3) using a deep reinforcement learning algorithm to train the pathfinding AI partner, so that it can autonomously explore and learn pathfinding in the game; (4) using a course learning method to gradually expand the area that the AI ​​can explore, avoiding the AI ​​partner being unable to learn pathfinding due to the task setting being too difficult.

[0075] In summary, the virtual character control method provided by this invention significantly improves the pathfinding reach rate compared to traditional solutions in complex 3D scenes. It achieves a pathfinding reach rate of over 90% when setting any distant target within the game. Furthermore, the pathfinding AI companion can autonomously use special in-game items, equipment, and mechanics to move, exhibiting a degree of anthropomorphism. Compared to existing technical solutions, the pathfinding AI companion in this invention demonstrates substantial improvements in both reach rate and performance. Specifically: (1) High pathfinding success rate: By introducing reinforcement learning algorithms, the average success rate of AI pathfinding has increased from 59% to 86%. More importantly, at some difficult target points, the pathfinding success rate has increased from 37% to 74%, achieving a doubling of performance; (2) High degree of human-like behavior: In addition to the significant improvement in quantitative indicators, the pathfinding AI trained by reinforcement learning algorithms will be closer to humans in terms of behavior patterns. It can skillfully use basic behaviors such as sprinting and jumping, can autonomously adjust its perspective, and can reasonably use props to quickly approach the target point when crossing ditches and climbing buildings. The reinforcement learning AI has even learned advanced operations such as charging slides and sliding hops.

[0076] Regarding the virtual character control method provided in the foregoing embodiments, this invention provides a virtual character control device, see [link to previous document]. Figure 6 The diagram shows a structural schematic of a control device for a virtual character. The device mainly includes the following parts:

[0077] The acquisition module 602 is used to acquire the target voxel data of the three-dimensional virtual scene in which the virtual character is located, as well as the state parameters of the virtual character in the three-dimensional virtual scene;

[0078] The action determination module 604 is used to determine the action to be executed based on target voxel data and state parameters through a target pathfinding model; wherein, the target training model is trained using a reinforcement learning algorithm;

[0079] The control module 606 is used to control the movement of a virtual character in a three-dimensional virtual scene based on the action to be performed.

[0080] The virtual character control device provided in this embodiment of the invention can determine the action to be executed based on the target voxel data and state parameters of the three-dimensional virtual scene through the target pathfinding model. The target voxel data enables the virtual character to have a more refined perception of the three-dimensional virtual scene, and on this basis, controls the virtual character to move in the three-dimensional virtual scene, thereby significantly improving the pathfinding success rate and also improving the anthropomorphism of the virtual character.

[0081] In one implementation, the state parameters include the current position parameter and the target position parameter; the action determination module 604 is further configured to: acquire a depth image of the three-dimensional virtual scene and the historical action corresponding to the previous position parameter of the current position parameter; and determine the action to be executed corresponding to the current position parameter based on the target voxel data, state parameters, depth image and historical action through the target pathfinding model.

[0082] In one implementation, the target pathfinding model includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to a processing sub-network. The action determination module 604 is further configured to: use target voxel data, state parameters, depth images, and historical actions as input data for each input terminal; extract feature information from the input data of each input terminal through each processing sub-network, and perform feature concatenation processing on each feature information to obtain concatenated feature information; determine the output data of each output terminal based on the concatenated feature information through the multilayer perceptron, and use the output data of each output terminal as the action to be executed.

[0083] In one embodiment, the input terminal includes a voxel input terminal, a state input terminal, a depth input terminal, and an action input terminal. The processing subnetwork includes a voxel processing subnetwork, a fully connected subnetwork, a residual subnetwork, and a Transformer subnetwork. The voxel input terminal is connected to the voxel processing subnetwork, the state input terminal is connected to the fully connected subnetwork, the depth input terminal is connected to the residual subnetwork, and the action input terminal is connected to the Transformer subnetwork. The action determination module 604 is further configured to: extract feature information of the target voxel data through the voxel processing subnetwork; extract feature information of the state parameters through the fully connected subnetwork; extract feature information of the depth image through the residual subnetwork; and extract feature information of historical actions through the Transformer subnetwork.

[0084] In one embodiment, the above-mentioned apparatus further includes a training module, configured to: acquire multiple first scene sets and second scene sets; wherein the scene complexity of the second scene set is greater than that of the first scene set; use a reinforcement learning algorithm to perform a first-stage training on an initial pathfinding model based on the first scene set until the initial pathfinding model meets a first preset requirement; and use a reinforcement learning algorithm to perform a second-stage training on the initial pathfinding model that meets the first preset requirement based on the second scene set until the initial pathfinding model meets the second preset requirement to obtain a target pathfinding model.

[0085] In one implementation, the training module is further configured to: use the first voxel data of the first preset scene in the first scene set as the model state of the initial pathfinding model, and use the first action of the virtual character in the first preset scene as the model action of the initial pathfinding model; determine the reward function corresponding to the initial pathfinding model, and use a reinforcement learning algorithm to perform a first stage of training on the initial pathfinding model based on the model state, model action and reward function.

[0086] In one implementation, the training module is further configured to: determine an intermediate reward based on the current point of the virtual character in the first preset scene, the previous point corresponding to the current point, and the target point; or, determine a time penalty based on the movement time taken by the virtual character to move from the previous point to the current point; or, determine a sparse reward based on the initial point and the target point of the virtual character in the first preset scene; and determine one or more of the intermediate reward, time penalty, and sparse reward as the reward function corresponding to the initial pathfinding model.

[0087] In one implementation, the action to be performed includes character movement parameters, which include the character movement direction and the character behavior action; the control module 606 is also used to: trigger the virtual character to perform the character behavior action and move in the three-dimensional virtual scene according to the character movement direction.

[0088] In one embodiment, the action to be performed further includes camera movement parameters, which include the horizontal movement direction of the camera and the vertical movement direction of the camera; the control module 606 is also used to control the virtual camera of the three-dimensional virtual scene to move in the three-dimensional virtual scene according to the horizontal movement direction of the camera and the vertical movement direction of the camera.

[0089] In one embodiment, the acquisition module 602 is further configured to: acquire initial voxel data of the three-dimensional virtual scene in which the virtual character is located; assign values ​​to each element in the initial voxel data according to the material of the objects contained in the three-dimensional virtual scene to obtain target voxel data; wherein, the elements in the target voxel data are used to characterize the material of the object corresponding to the element; or, associate the initial voxel data with the material of the objects contained in the three-dimensional virtual scene respectively, and for each material, assign values ​​to the elements in the initial voxel data associated with the material that match the material to obtain the target voxel data associated with the material.

[0090] This disclosure provides a terminal device, specifically, the terminal device includes a processor and a storage device; the storage device stores a computer program, which is executed by the processor when it runs:

[0091] The system acquires target voxel data of the three-dimensional virtual scene in which the virtual character is located, as well as the state parameters of the virtual character in the three-dimensional virtual scene; it determines the action to be executed based on the target voxel data and state parameters through a target pathfinding model; wherein the target training model is trained using a reinforcement learning algorithm; and it controls the virtual character to move in the three-dimensional virtual scene based on the action to be executed.

[0092] In one implementation, the state parameters include current position parameters and target position parameters; the step of determining the action to be executed based on target voxel data and state parameters through a target pathfinding model includes: acquiring a depth image of the three-dimensional virtual scene and historical actions corresponding to the previous position parameter of the current position parameter; and determining the action to be executed corresponding to the current position parameter based on target voxel data, state parameters, depth image and historical actions through a target pathfinding model.

[0093] In one implementation, the target pathfinding model includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to a processing sub-network. The step of determining the action to be executed corresponding to the current position parameters based on target voxel data, state parameters, depth image, and historical actions using the target pathfinding model includes: using the target voxel data, state parameters, depth image, and historical actions as input data for each input terminal; extracting feature information from the input data of each input terminal through each processing sub-network, and performing feature concatenation on each feature information to obtain concatenated feature information; and determining the output data of each output terminal based on the concatenated feature information using the multilayer perceptron, and using the output data of each output terminal as the action to be executed.

[0094] In one embodiment, the input terminals include a voxel input terminal, a state input terminal, a depth input terminal, and an action input terminal. The processing subnetwork includes a voxel processing subnetwork, a fully connected subnetwork, a residual subnetwork, and a Transformer subnetwork. The voxel input terminal is connected to the voxel processing subnetwork, the state input terminal is connected to the fully connected subnetwork, the depth input terminal is connected to the residual subnetwork, and the action input terminal is connected to the Transformer subnetwork. The step of extracting feature information of the input data of each input terminal through each processing subnetwork includes: extracting feature information of target voxel data through the voxel processing subnetwork; extracting feature information of state parameters through the fully connected subnetwork; extracting feature information of depth images through the residual subnetwork; and extracting feature information of historical actions through the Transformer subnetwork.

[0095] In one embodiment, the method further includes: acquiring multiple first scene sets and second scene sets; wherein the scene complexity of the second scene set is greater than that of the first scene set; using a reinforcement learning algorithm, performing a first-stage training on an initial pathfinding model based on the first scene set until the initial pathfinding model meets a first preset requirement; and using a reinforcement learning algorithm, performing a second-stage training on the initial pathfinding model that meets the first preset requirement based on the second scene set until the initial pathfinding model meets the second preset requirement to obtain a target training model.

[0096] In one implementation, the step of training an initial pathfinding model in the first stage based on a first scene set includes: using the first voxel data of a first preset scene in the first scene set as the model state of the initial pathfinding model, and using the first action of a virtual character in the first preset scene as the model action of the initial pathfinding model; determining the reward function corresponding to the initial pathfinding model, and using a reinforcement learning algorithm to train the initial pathfinding model in the first stage based on the model state, model action, and reward function.

[0097] In one implementation, the step of determining the reward function corresponding to the initial pathfinding model includes: determining an intermediate reward based on the current point of the virtual character in the first preset scene, the previous point corresponding to the current point, and the target point; or determining a time penalty based on the movement time taken by the virtual character to move from the previous point to the current point; or determining a sparse reward based on the initial point and the target point of the virtual character in the first preset scene; and determining one or more of the intermediate reward, time penalty, and sparse reward as the reward function corresponding to the initial pathfinding model.

[0098] In one implementation, the action to be executed includes character movement parameters, which include the character movement direction and the character behavior action; the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be executed includes: triggering the virtual character to execute the character behavior action, and moving in the three-dimensional virtual scene according to the character movement direction.

[0099] In one embodiment, the action to be performed further includes camera movement parameters, which include a horizontal camera movement direction and a vertical camera movement direction; after the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be performed, the method further includes: controlling the virtual camera of the three-dimensional virtual scene to move in the three-dimensional virtual scene according to the horizontal camera movement direction and the vertical camera movement direction.

[0100] In one implementation, the step of obtaining target voxel data of a three-dimensional virtual scene in which a virtual character is located includes: obtaining initial voxel data of the three-dimensional virtual scene in which the virtual character is located; assigning values ​​to each element in the initial voxel data according to the material of the objects contained in the three-dimensional virtual scene to obtain target voxel data; wherein, the elements in the target voxel data are used to characterize the material of the object corresponding to the element; or, associating the initial voxel data with the material of the objects contained in the three-dimensional virtual scene respectively, and for each material, assigning values ​​to the elements in the initial voxel data associated with the material that match the material to obtain the target voxel data associated with the material.

[0101] The terminal device provided in this embodiment of the invention can determine the action to be executed based on the target voxel data and state parameters of the three-dimensional virtual scene through the target pathfinding model. The target voxel data enables the virtual character to have a more refined perception of the three-dimensional virtual scene, and on this basis, controls the virtual character to move in the three-dimensional virtual scene, thereby significantly improving the pathfinding success rate and also improving the anthropomorphism of the virtual character.

[0102] The specific implementation details of the computer program running in this embodiment are also applicable to the aforementioned implementation details of the virtual character control method, so they will not be repeated here.

[0103] This invention provides a terminal device, specifically, the terminal device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0104] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. The terminal device 100 includes: a processor 70, a memory 71, a bus 72 and a communication interface 73. The processor 70, the communication interface 73 and the memory 71 are connected through the bus 72. The processor 70 is used to execute executable modules, such as computer programs, stored in the memory 71.

[0105] The memory 71 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 73 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0106] Bus 72 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0107] The memory 71 is used to store programs. After receiving an execution instruction, the processor 70 executes the programs. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 70 or implemented by the processor 70.

[0108] The processor 70 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 70 or by instructions in software form. The processor 70 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 71. Processor 70 reads the information in memory 71 and, in conjunction with its hardware, completes the steps of the above method.

[0109] The computer program product of the readable storage medium provided in this disclosure includes a computer-readable storage medium storing program code, the program code including instructions that can be executed:

[0110] The system acquires target voxel data of the three-dimensional virtual scene in which the virtual character is located, as well as the state parameters of the virtual character in the three-dimensional virtual scene; it determines the action to be executed based on the target voxel data and state parameters through a target pathfinding model; wherein the target training model is trained using a reinforcement learning algorithm; and it controls the virtual character to move in the three-dimensional virtual scene based on the action to be executed.

[0111] In one implementation, the state parameters include current position parameters and target position parameters; the step of determining the action to be executed based on target voxel data and state parameters through a target pathfinding model includes: acquiring a depth image of the three-dimensional virtual scene and historical actions corresponding to the previous position parameter of the current position parameter; and determining the action to be executed corresponding to the current position parameter based on target voxel data, state parameters, depth image and historical actions through a target pathfinding model.

[0112] In one implementation, the target pathfinding model includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to a processing sub-network. The step of determining the action to be executed corresponding to the current position parameters based on target voxel data, state parameters, depth image, and historical actions using the target pathfinding model includes: using the target voxel data, state parameters, depth image, and historical actions as input data for each input terminal; extracting feature information from the input data of each input terminal through each processing sub-network, and performing feature concatenation on each feature information to obtain concatenated feature information; and determining the output data of each output terminal based on the concatenated feature information using the multilayer perceptron, and using the output data of each output terminal as the action to be executed.

[0113] In one embodiment, the input terminals include a voxel input terminal, a state input terminal, a depth input terminal, and an action input terminal. The processing subnetwork includes a voxel processing subnetwork, a fully connected subnetwork, a residual subnetwork, and a Transformer subnetwork. The voxel input terminal is connected to the voxel processing subnetwork, the state input terminal is connected to the fully connected subnetwork, the depth input terminal is connected to the residual subnetwork, and the action input terminal is connected to the Transformer subnetwork. The step of extracting feature information of the input data of each input terminal through each processing subnetwork includes: extracting feature information of target voxel data through the voxel processing subnetwork; extracting feature information of state parameters through the fully connected subnetwork; extracting feature information of depth images through the residual subnetwork; and extracting feature information of historical actions through the Transformer subnetwork.

[0114] In one embodiment, the method further includes: acquiring multiple first scene sets and second scene sets; wherein the scene complexity of the second scene set is greater than that of the first scene set; using a reinforcement learning algorithm, performing a first-stage training on an initial pathfinding model based on the first scene set until the initial pathfinding model meets a first preset requirement; and using a reinforcement learning algorithm, performing a second-stage training on the initial pathfinding model that meets the first preset requirement based on the second scene set until the initial pathfinding model meets the second preset requirement to obtain a target training model.

[0115] In one implementation, the step of using a reinforcement learning algorithm to perform a first-stage training of an initial pathfinding model based on a first scene set includes: using the first voxel data of a first preset scene in the first scene set as the model state of the initial pathfinding model, and using the first action of a virtual character in the first preset scene as the model action of the initial pathfinding model; determining the reward function corresponding to the initial pathfinding model, and using a reinforcement learning algorithm to perform a first-stage training of the initial pathfinding model based on the model state, model action, and reward function.

[0116] In one implementation, the step of determining the reward function corresponding to the initial pathfinding model includes: determining an intermediate reward based on the current point of the virtual character in the first preset scene, the previous point corresponding to the current point, and the target point; or determining a time penalty based on the movement time taken by the virtual character to move from the previous point to the current point; or determining a sparse reward based on the initial point and the target point of the virtual character in the first preset scene; and determining one or more of the intermediate reward, time penalty, and sparse reward as the reward function corresponding to the initial pathfinding model.

[0117] In one implementation, the action to be executed includes character movement parameters, which include the character movement direction and the character behavior action; the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be executed includes: triggering the virtual character to execute the character behavior action, and moving in the three-dimensional virtual scene according to the character movement direction.

[0118] In one embodiment, the action to be performed further includes camera movement parameters, which include a horizontal camera movement direction and a vertical camera movement direction; after the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be performed, the method further includes: controlling the virtual camera of the three-dimensional virtual scene to move in the three-dimensional virtual scene according to the horizontal camera movement direction and the vertical camera movement direction.

[0119] In one implementation, the step of obtaining target voxel data of a three-dimensional virtual scene in which a virtual character is located includes: obtaining initial voxel data of the three-dimensional virtual scene in which the virtual character is located; assigning values ​​to each element in the initial voxel data according to the material of the objects contained in the three-dimensional virtual scene to obtain target voxel data; wherein, the elements in the target voxel data are used to characterize the material of the object corresponding to the element; or, associating the initial voxel data with the material of the objects contained in the three-dimensional virtual scene respectively, and for each material, assigning values ​​to the elements in the initial voxel data associated with the material that match the material to obtain the target voxel data associated with the material.

[0120] The readable storage medium provided in this embodiment of the invention can determine the action to be executed based on the target voxel data and state parameters of the three-dimensional virtual scene through the target pathfinding model. The target voxel data can enable the virtual character to have a more refined perception of the three-dimensional virtual scene, and on this basis, control the virtual character to move in the three-dimensional virtual scene, thereby significantly improving the pathfinding success rate and also improving the anthropomorphism of the virtual character.

[0121] The specific implementation details of the program code used in this embodiment are also applicable to the aforementioned implementation details of the virtual character control method, and therefore will not be repeated here.

[0122] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling a virtual character, characterized in that, include: Acquire the initial voxel data of the three-dimensional virtual scene in which the virtual character is located, as well as the state parameters of the virtual character in the three-dimensional virtual scene; Based on the material of the objects contained in the 3D virtual scene, each element in the initial voxel data is assigned a value to obtain target voxel data; wherein, the elements in the target voxel data are used to represent the material of the object corresponding to the element; or, the initial voxel data is associated with the material of the objects contained in the 3D virtual scene respectively, and for each material, the element in the initial voxel data associated with that material that matches the material is assigned a value to obtain the target voxel data associated with that material; the state parameters include current position parameters and target position parameters; The depth image of the 3D virtual scene is acquired, along with the historical actions corresponding to the previous position parameter of the current position parameter. Based on the target voxel data, the state parameters, the depth image, and the historical actions, the target pathfinding model determines the action to be executed corresponding to the current position parameter. The target training model is trained using a reinforcement learning algorithm. Based on the action to be performed, control the virtual character to move within the three-dimensional virtual scene; The target pathfinding model includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to one of the processing sub-networks. The step of determining the action to be executed corresponding to the current position parameter based on the target voxel data, the state parameters, the depth image, and the historical actions using the target pathfinding model includes: using the target voxel data, the state parameters, the depth image, and the historical actions as input data for each input terminal; extracting feature information from the input data of each input terminal through each processing sub-network, and performing feature concatenation processing on each feature information to obtain concatenated feature information; determining the output data of each output terminal based on the concatenated feature information using the multilayer perceptron, and using the output data of each output terminal as the action to be executed.

2. The method according to claim 1, characterized in that, The input terminals include a voxel input terminal, a state input terminal, a depth input terminal, and an action input terminal. The processing sub-network includes a voxel processing sub-network, a fully connected sub-network, a residual sub-network, and a Transformer sub-network. The voxel input terminal is connected to the voxel processing sub-network, the state input terminal is connected to the fully connected sub-network, the depth input terminal is connected to the residual sub-network, and the action input terminal is connected to the Transformer sub-network. The step of extracting feature information of the input data from each input terminal through each processing sub-network includes: Feature information of the target voxel data is extracted through the voxel processing subnetwork; feature information of the state parameters is extracted through the fully connected subnetwork; feature information of the depth image is extracted through the residual subnetwork; and feature information of the historical actions is extracted through the Transformer subnetwork.

3. The method according to claim 1, characterized in that, The method further includes: Obtain multiple first scene sets and second scene sets; wherein the scene complexity of the second scene set is greater than the scene complexity of the first scene set; Using the reinforcement learning algorithm, the initial pathfinding model is trained in the first stage based on the first scene set until the initial pathfinding model meets the first preset requirement; Using the reinforcement learning algorithm, the initial pathfinding model that meets the first preset requirement is trained in the second stage based on the second scene set until the initial pathfinding model meets the second preset requirement to obtain the target pathfinding model.

4. The method according to claim 3, characterized in that, The step of using the reinforcement learning algorithm to perform the first stage training of the initial pathfinding model based on the first scene set includes: The first voxel data of the first preset scene in the first scene set is used as the model state of the initial pathfinding model, and the first action of the virtual character in the first preset scene is used as the model action of the initial pathfinding model. The reward function corresponding to the initial pathfinding model is determined, and the reinforcement learning algorithm is used to perform the first stage training of the initial pathfinding model based on the model state, the model action, and the reward function.

5. The method according to claim 4, characterized in that, The step of determining the reward function corresponding to the initial pathfinding model includes: An intermediate reward is determined based on the virtual character's current point in the first preset scene, the previous point corresponding to the current point, and the target point; or, a time penalty is determined based on the movement time taken by the virtual character to move from the previous point to the current point; or, a sparse reward is determined based on the virtual character's initial point and the target point in the first preset scene. One or more of the intermediate reward, the time penalty, and the sparse reward are determined as the reward function corresponding to the initial pathfinding model.

6. The method according to claim 1, characterized in that, The action to be executed includes character movement parameters, which include the character's movement direction and character behavior; the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be executed includes: The virtual character is triggered to perform the character's actions and move in the three-dimensional virtual scene according to the character's movement direction.

7. The method according to claim 6, characterized in that, The action to be executed further includes camera movement parameters, which include the camera's horizontal movement direction and the camera's vertical movement direction; after the step of controlling the virtual character to move in the three-dimensional virtual scene based on the action to be executed, the method further includes: The virtual camera controlling the three-dimensional virtual scene moves within the three-dimensional virtual scene according to the camera's horizontal and vertical movement directions.

8. A control device for a virtual character, characterized in that, include: The acquisition module is used to acquire target voxel data of the three-dimensional virtual scene in which the virtual character is located, as well as the state parameters of the virtual character in the three-dimensional virtual scene; An action determination module is used to determine the action to be performed based on the target voxel data and the state parameters using a target pathfinding model; wherein the target training model is trained using a reinforcement learning algorithm. The control module is used to control the virtual character to move in the three-dimensional virtual scene based on the action to be performed; The acquisition module is specifically used to: acquire the initial voxel data of the 3D virtual scene in which the virtual character is located; assign values ​​to each element in the initial voxel data according to the material of the objects contained in the 3D virtual scene to obtain the target voxel data; wherein, the elements in the target voxel data are used to represent the material of the corresponding object; or, associate the initial voxel data with the material of the objects contained in the 3D virtual scene respectively, and for each material, assign values ​​to the elements in the initial voxel data associated with that material that match that material to obtain the target voxel data associated with that material; The state parameters include the current position parameters and the target position parameters; the action determination module is also used to: acquire the depth image of the 3D virtual scene and the historical actions corresponding to the previous position parameters of the current position parameters; and determine the action to be executed corresponding to the current position parameters based on the target voxel data, state parameters, depth image and historical actions through the target pathfinding model. The target pathfinding model includes multiple input terminals, multiple processing sub-networks, a multilayer perceptron, and multiple output terminals, with each input terminal corresponding to a processing sub-network. The action determination module is further configured to: use target voxel data, state parameters, depth images, and historical actions as input data for each input terminal; extract feature information from the input data of each input terminal through each processing sub-network, and perform feature concatenation processing on each feature information to obtain concatenated feature information; and determine the output data of each output terminal based on the concatenated feature information using the multilayer perceptron, and use the output data of each output terminal as the action to be executed.

9. A terminal device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images