Virtual scene path finding method, device, electronic device and storage medium

Abstract

The present application provides a virtual scene pathfinding method and related equipment, including: obtaining the current position and destination position of the virtual object; inputting the current position and destination position into the path midpoint prediction model to calculate the intermediate node position, and using the obtained intermediate node position as the target in the virtual object reinforcement learning process; using the intermediate node position as the next pathfinding target position of the virtual object, and controlling the virtual object to move to the pathfinding target position. The present application simulates the user's real movement route and movement operation in the virtual scene through the learning of the intermediate point prediction model, and divides the long-distance pathfinding into multiple short-distance pathfinding segments by setting up intermediate nodes, strengthens the route flexibility during pathfinding, and makes the automatic pathfinding in the virtual scene more like the movement route operated by the user himself, thereby providing a more normal and anthropomorphic automatic pathfinding solution, improving the pathfinding efficiency and pathfinding success rate while improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a virtual scene path-finding method, device, electronic device and storage medium. Background Art

[0002] In the current use of more and more game applications or interactive applications, in order to adapt to the needs of actual applications, it is possible to realize diversified movements of virtual objects controlled by users or artificial intelligence according to actual scenarios, for example: automatic pathfinding solutions for virtual objects in virtual scenes.

[0003] However, some existing automatic path-finding solutions generally control the movement of virtual objects by using scripts written according to certain rules. This method is difficult to write rules, and in some scenarios, the automatic path-finding movement of virtual objects is significantly different from the movement of virtual objects controlled by normal users, and the behavior is rather strange. Summary of the invention

[0004] In view of this, the present application proposes a virtual scene path-finding method, device, electronic device and storage medium, so as to provide a more normal and humanized automatic path-finding solution.

[0005] Based on the above purpose, the present application provides a pathfinding method for a virtual scene, comprising:

[0006] Obtaining a current position and a destination position of a virtual object, wherein the virtual object is an intelligent agent obtained through reinforcement learning;

[0007] Inputting the current position and the destination position into a path midpoint prediction model to calculate an intermediate node position, and using the obtained intermediate node position as a target in the virtual object reinforcement learning process;

[0008] The intermediate node position is used as the next pathfinding target position of the virtual object, and the virtual object is controlled to move toward the pathfinding target position.

[0009] In some embodiments, the method further comprises:

[0010] Determine a virtual scene where the virtual object is currently located, and obtain at least one piece of historical movement data of the virtual object controlled by the user in the virtual scene;

[0011] Generate a training set according to the historical movement data, each data of the training set includes a starting point and an end point corresponding to each movement step, and a label composed of a next point of the starting point, wherein the next point is a point on a path formed by the starting point and the end point;

[0012] The path midpoint prediction model is trained using the training set.

[0013] In some implementations, generating a training set according to the historical movement data includes:

[0014] generating a visual path map according to the historical movement data;

[0015] Cleaning the historical movement data according to the visualized path map;

[0016] A training set is generated according to the cleaned historical movement data.

[0017] In some implementations, generating a visual path map according to the historical movement data includes:

[0018] Using different styles to display the movement paths of different players, the movement paths forming the visual path map;

[0019] Cleaning the historical movement data according to the visualized path map includes: if there is an aggregation between two movement paths of different styles in the visualized path map, deleting the historical movement data corresponding to the aggregation area of ​​the two movement paths of different styles from the historical movement data; or / and,

[0020] If the corresponding displacement of a single moving path in the visual path diagram is less than a preset distance, the single moving path is deleted from the historical movement data.

[0021] In some implementations, the training of the path midpoint prediction model using the training set includes:

[0022] Inputting the first data in the training set into the path midpoint prediction model, and outputting the next point predicted by the path midpoint prediction model;

[0023] Comparing the predicted next point with the label in the first piece of data, and determining whether a preset convergence condition is met according to the comparison result, and if not, adjusting the parameters of the path midpoint prediction model based on the comparison result;

[0024] The path midpoint prediction model after parameter adjustment is trained for the next round based on the training set until a preset convergence condition is met, thereby obtaining a trained path midpoint prediction model.

[0025] In some embodiments, the method further comprises:

[0026] Acquiring current state data of the object and action data that the virtual object can perform;

[0027] Reinforcement learning is performed on the virtual object based on the state data, the action data and the next node predicted by the path midpoint prediction model, and the next round of reinforcement learning is performed on the virtual object according to the reward obtained from the reinforcement learning until a preset convergence condition is met to obtain a trained virtual object.

[0028] In some implementations, after controlling the virtual object to move toward the pathfinding target position, the method further includes:

[0029] In response to the virtual object reaching the pathfinding target position within a set number of steps, the pathfinding target position is used as the new current position of the virtual object, the pathfinding target position is redetermined based on the new current position, and the virtual object is controlled to move toward the redetermined pathfinding target position.

[0030] In some implementations, after controlling the virtual object to move toward the pathfinding target position, the method further includes:

[0031] In response to the virtual object not reaching the pathfinding target position within a set number of steps, the current position of the virtual object is reacquired, the pathfinding target position is re-determined based on the re-acquired current position, and the virtual object is controlled to move toward the re-determined pathfinding target position.

[0032] In some embodiments, the path midpoint prediction model is a fully connected neural network model.

[0033] Based on the same concept, the present application also provides a virtual scene pathfinding device, including:

[0034] An acquisition module, used to acquire the current position and the destination position of a virtual object, wherein the virtual object is an intelligent agent obtained through reinforcement learning;

[0035] A determination module, used for inputting the current position and the destination position into a path midpoint prediction model to calculate an intermediate node position, and using the obtained intermediate node position as a target in the virtual object reinforcement learning process;

[0036] The control module is used to use the intermediate node position as the next pathfinding target position of the virtual object and control the virtual object to move toward the pathfinding target position.

[0037] Based on the same concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0038] Based on the same concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to implement any of the methods described above.

[0039] From the above, it can be seen that the present application provides a virtual scene pathfinding method, device, electronic device and storage medium, including: obtaining the current position and destination position of a virtual object, wherein the virtual object is an intelligent body obtained through reinforcement learning; inputting the current position and the destination position into the path midpoint prediction model to calculate the intermediate node position, and using the obtained intermediate node position as the target in the virtual object reinforcement learning process; using the intermediate node position as the next pathfinding target position of the virtual object, and controlling the virtual object to move to the pathfinding target position. The present application simulates the user's real moving route and moving operation in the virtual scene through the learning of the intermediate point prediction model, and divides the long-distance pathfinding into multiple short-distance pathfinding by setting up intermediate nodes, strengthens the route flexibility during pathfinding, and makes the automatic pathfinding in the virtual scene more like the moving route operated by the user himself, thereby providing a more normal and anthropomorphic automatic pathfinding solution, which improves the pathfinding efficiency and pathfinding success rate while improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a path-finding method for a virtual scene proposed in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the structure of a virtual scene path-finding device proposed in an embodiment of the present application;

[0043] Figure 3 This is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this specification more clear, this specification is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0045] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements, objects or method steps appearing before the word cover the elements, objects or method steps listed after the word and their equivalents, without excluding other elements, objects or method steps. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] As described in the background technology section, taking the specific scene of game application as an example, currently many pathfinding robots in many games are scripts, that is, written according to certain rules. It is difficult to write rules in this way, and it seems that many behavior models are contrary to the norm and undesirable behaviors (that is, behaviors that are not very human-controlled). For example, the current target of the pathfinding robot is the player closest to it, but when the player is outside the scope of the rules, it will also run towards the player. This behavior is very strange. Generally, the virtual objects manipulated by normal users will not actively run outside the scope of the rules; in addition, there are many houses in the game scene, and there are supplies in the houses. The virtual objects manipulated by normal users will go out of the door or jump out of the window after taking the supplies, but the pathfinding robot may be stuck in a corner of the room and cannot get out, and will constantly try repeated actions such as jumping forward, etc., eager to go out but still stuck; when encountering a high wall with an eaves, the virtual object manipulated by the normal user will use a hook lock to flip over when it is far away from the wall, and the pathfinding robot may walk to the front of the wall and try to jump up and flip over, but it is blocked by the eaves and cannot flip over.

[0047] In some existing improvement schemes, a pathfinding robot is directly trained using a reinforcement learning model, with the game scene as the environment, the observed game scene and state as the state, and the predicted series of actions as the action, and the training is carried out with the goal of reaching the predicted target point. The trained pathfinding robot can reach the target point as much as possible, and can also stay away from the boundaries of the rule range to a certain extent, but there are still cases of getting stuck. The main reason for the stuck is that the pathfinding robot reaches the destination by exploring the path itself, unlike human behavior, which is easy to get stuck. For example, in a hall, the first floor is full of doors, and the windows on the second floor are closed. Normal players will leave the room from the first floor door, but the pathfinding robot may try to jump to the second floor and want to go out through the window, but it will get stuck if it can't get out. That is, the pathfinding method of the existing scheme is still relatively low in anthropomorphism and poor in flexibility, which ultimately leads to low pathfinding efficiency and pathfinding success rate.

[0048] In view of the above-mentioned actual situation, the embodiment of the present application proposes a pathfinding solution for a virtual scene. The intermediate point prediction model is used to learn and simulate the user's real movement route and movement operation in the virtual scene, and the long-distance pathfinding is divided into multiple short-distance pathfinding segments by setting up intermediate nodes. The route flexibility during pathfinding is strengthened, and the automatic pathfinding in the virtual scene is more like the movement route operated by the user himself. This provides a more normal and humanized automatic pathfinding solution, which improves the pathfinding efficiency and pathfinding success rate while improving the user experience.

[0049] like Figure 1 FIG. 1 is a flow chart of a virtual scene path finding method proposed in the present application, and the method specifically includes:

[0050] Step 101, obtaining the current position and the destination position of a virtual object, wherein the virtual object is an intelligent agent obtained through reinforcement learning.

[0051] In this step, the virtual object is a virtual character object manipulated by the user or controlled by artificial intelligence in the virtual scene, and the current position information of the virtual object is obtained, and the destination position information of the virtual object is determined. Among them, the current position information and the destination position information can be coordinate information in a three-dimensional coordinate system established with the virtual scene as the background, or corresponding information that can reflect the specific position determined according to the specific application scenario. Afterwards, the destination position can be specified by the user when manipulating the virtual object, or it can be set for the virtual object through artificial intelligence according to specific rules. The method of determining the destination position can be different according to the specific application scenario.

[0052] In a specific embodiment, the current position and the target position are the starting position and the ending position of the path finding, and the goal of the path finding is to move the virtual object from the starting position to the ending position.

[0053] Step 102: Input the current position and the destination position into a path midpoint prediction model to calculate an intermediate node position, and use the obtained intermediate node position as a target in the virtual object reinforcement learning process.

[0054] In this step, an intermediate node position is first determined through the trained path intermediate point prediction model, and the virtual object finally reaches the destination position by reaching the intermediate node positions one by one.

[0055] Among them, the path midpoint prediction model can essentially be a relatively conventional neural network model, such as a fully connected neural network model, a convolutional neural network model, etc. The input dimension is 2*game coordinate dimension (i.e., current position and destination position), and the output dimension is the game coordinate dimension (i.e., the intermediate node position). The number and size of the intermediate neural network layers can be specifically designed according to the specific application scenario. The loss function (Loss) can use L2 loss (MeanSquared Error, MSE, mean square error), etc. The training method is the ordinary supervised learning method for training.

[0056] As for the training set of the model, it can be obtained by obtaining the historical records of the user's real mobile operations in the current virtual scene, etc., and the starting point and end point of each path in these real records are used as the input of the training set, that is, the current position and the destination position, and then these paths are segmented according to the preset length, the preset length is the same as the distance from the intermediate node position to the current position, and the first segmentation point after segmentation is used as the output with the starting point and the end point as the current position and the destination position, and the second segmentation point is used as the output with the first segmentation point and the end point as the current position and the destination position, and so on. After collecting a large number of historical records of mobile operations, these historical records are used as training sets. Of course, in specific application scenarios, the training set can also be obtained in other ways, such as specifying or making a fixed training set by the operator, or directly setting the various parameters and loss functions of the intermediate point prediction model. Afterwards, in some embodiments, the historical records of the user's real mobile operations can also be optimized and screened, so that these historical records are more in line with the requirements of path finding. Finally, the real path data of the real user collected is used as the training set for training, so that the intermediate point prediction model can plan a path from the starting point to the end point according to the starting point and the end point. Since the dataset is real user data, the planned path is more like the path that real users would take.

[0057] In a specific embodiment, the intermediate point prediction model predicts the next coordinate point, ie, the intermediate node position, based on the current position and the destination position of the robot, so that the virtual object can move toward the predicted coordinate point as the current target point.

[0058] Step 103: Use the intermediate node position as the next pathfinding target position of the virtual object, and control the virtual object to move toward the pathfinding target position.

[0059] In this step, after the intermediate node position is determined, the virtual object can use the intermediate node position as the current pathfinding target position, which is the target point position to which the current virtual object is to move, and is a section of the route planned from the current position to the destination position. In a specific embodiment, when the virtual object moves to the pathfinding target position, the next pathfinding target position can be calculated again according to this solution.

[0060] Afterwards, the virtual object is controlled to move toward the pathfinding target position, and the behavior actions used for its movement can also be determined by a neural network model, such as training through a reinforcement learning model, by obtaining scene information of the current virtual scene (which may include road information, forest information, house information, etc. in the virtual scene), and action information that the virtual object can use, such as in some scenes the virtual character can run, jump, climb, use specific props, etc., while in some virtual scenes the virtual character can only walk, squat, etc. A series of behavior actions are output, and the virtual object moves toward the pathfinding target position through these behavior actions.

[0061] In addition, the determined pathfinding target position can also be output for storing, displaying, using or reprocessing the pathfinding target position. For example, the current pathfinding target position is output and displayed in a display manner so that the user can intuitively observe the moving target of the virtual object, or the pathfinding target position is output to a downstream processing unit for further processing, such as the downstream processing unit generating an indication arrow based on the pathfinding target position and attaching it to the virtual object and displaying it so that the user can intuitively observe the moving direction of the current virtual object, etc. That is, the specific output method for the pathfinding target position can be flexibly selected according to different application scenarios and implementation needs.

[0062] For example, for an application scenario in which the method of this embodiment is executed on a single device, the path finding target location can be directly output in a displayed manner on the display component (display, projector, etc.) of the current device, so that the operator of the current device can directly see the content of the path finding target location from the display component.

[0063] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the pathfinding target location can be sent to other preset devices as receivers in the system, that is, the synchronization terminal, through any data communication method (wired connection, NFC, Bluetooth, wifi, cellular mobile network, etc.), so that the synchronization terminal can perform subsequent processing on it. Optionally, the synchronization terminal can be a preset server, which is generally set up in the cloud as a data processing and storage center, which can store and distribute the pathfinding target location; wherein the recipient of the distribution is the terminal device, and the holder or operator of these terminal devices can be the current user, the game quality supervisor, the user behavior supervisor (conducting cheating behavior review, etc.), and so on.

[0064] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the path finding target location can be sent directly to a preset terminal device through any data communication method, and the terminal device can be one or more of the ones listed in the preceding paragraphs.

[0065] From the above, it can be seen that a pathfinding method for a virtual scene in an embodiment of the present application includes: obtaining the current position and the destination position of a virtual object, wherein the virtual object is an intelligent body obtained through reinforcement learning; inputting the current position and the destination position into a path midpoint prediction model to calculate the intermediate node position, and using the obtained intermediate node position as the target in the reinforcement learning process of the virtual object; using the intermediate node position as the next pathfinding target position of the virtual object, and controlling the virtual object to move toward the pathfinding target position. The present application simulates the user's real moving route and moving operation in the virtual scene through the learning of the intermediate point prediction model, and divides the long-distance pathfinding into multiple short-distance pathfinding segments by setting up intermediate nodes, strengthens the route flexibility during pathfinding, and makes the automatic pathfinding in the virtual scene more like the moving route operated by the user himself, thereby providing a more normal and anthropomorphic automatic pathfinding solution, which improves the pathfinding efficiency and pathfinding success rate while improving the user experience.

[0066] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of the embodiment of the present application can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0067] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0068] In an optional exemplary embodiment, before the current position and the destination position are input into the path midpoint prediction model to calculate the midpoint node position, it also includes: determining the virtual scene where the virtual object is currently located, obtaining at least one historical movement data of the virtual object controlled by the user in the virtual scene, and generating a training set based on the historical movement data, each data of the training data set contains the starting point and end point corresponding to each moving step, and a label composed of the next point of the starting point, wherein the next point is a point on the path formed by the starting point and the end point.

[0069] In this embodiment, the training set of the path midpoint prediction model is generated by collecting the mobile operation data of real users in this virtual scene and preprocessing these mobile operation data. Taking the game scene as an example, in some interactive and confrontational games, multiple map virtual scenes are generally set. Each time the game is played, it will be randomly or according to the user's designation to one of the map virtual scenes, and multiple users will interact or compete in the map virtual scene. Therefore, when the game is launched, many records of users playing the game can be recorded for a virtual scene, and the coordinate information of the user in the process of playing the game (i.e., historical movement data) is mainly extracted from these records. Through these coordinate information, the user's movement trajectory can be known, and according to the user's movement trajectory, the automatic path finding can imitate the user's movement trajectory to achieve more anthropomorphic movement. Afterwards, since the movement trajectory of the user when playing the game is generally not all path finding, it may be interacting or competing with other users, or it may be that he is standing still in some places or moving back and forth in a small range or turning in circles, etc., and these data are of no reference value for path finding, and all need to be eliminated. Therefore, it is necessary to screen these data, wherein the screening may be to clean the data. When preprocessing these historical mobile data, a corresponding targeted preprocessing process may be performed according to the specific application scenario.

[0070] In an optional exemplary embodiment, generating a training set based on the historical movement data includes: generating a visual path map based on the historical movement book; cleaning the historical movement data based on the visual path map; and generating a training set based on the cleaned historical movement data. In another optional exemplary embodiment, generating a visual path map based on the historical movement data includes: using different styles to display the movement paths of different players, and the movement paths form the visual path map. In another optional exemplary embodiment, cleaning the historical movement data includes: if there is an aggregation of two movement paths of different styles in the visual path map, deleting the historical movement data corresponding to the aggregation area of ​​the two movement paths of different styles from the historical movement data; or / and, if the corresponding displacement of a single movement path in the visual path map is less than a preset distance, deleting the single movement path from the historical movement data.

[0071] In an optional exemplary embodiment, the training of the path midpoint prediction model through the training set specifically includes: inputting the first data in the training set into the path midpoint prediction model, and outputting the next point predicted by the path midpoint prediction model; comparing the predicted next point with the label in the first data, and determining whether the preset convergence condition is met based on the comparison result, if not, adjusting the parameters of the path midpoint prediction model based on the comparison result; and performing the next round of training on the path midpoint prediction model after parameter adjustment based on the training set until the preset convergence condition is met, and obtaining the trained path midpoint prediction model. In these embodiments, the operation duration corresponds to the duration information of the user manipulating the virtual character. In some application scenarios, the user may only manipulate the virtual character for a very short time, for example, the user may fail or "die" in a very short time after entering the virtual scene and exit the virtual scene, and then these records with too short operation time have low reference value for path finding itself, and these historical movement data can be deleted. The moving distance corresponds to the distance that the user manipulates the virtual character to move in the virtual scene. In some application scenarios, the user may only manipulate the virtual character to move a short distance or not perform any moving operation at all. Therefore, the records of these operations with too short moving distances are also of low reference value to the pathfinding itself, and these historical movement data can be deleted and other operations. The movement mode corresponds to the way the user manipulates the virtual character to move in the virtual scene. Some users only move within a specific range or move in circles after entering the virtual scene. For example, after entering the virtual scene, the user only manipulates the virtual character to move in a house or spin in place. The records of these operation modes that are contrary to the purpose of pathfinding itself are also of low reference value to the pathfinding itself, and these historical movement data can be deleted and other operations. The above situation is to perform the deletion operation on the entire historical movement data, that is, the screening work. Afterwards, each piece of historical movement data can also be cleaned so that the data presented by it can better reflect pathfinding or movement.

[0072] In these embodiments, after entering the virtual scene, the user does not only move or find the way, but more importantly, operates the virtual objects in the virtual scene to confront or interact with the virtual objects manipulated by other users. However, the operation data of these confrontations or interactions themselves provide a low reference value for pathfinding, so they can be eliminated from the historical movement data. To determine whether a user has a confrontation or interaction, it can be determined that the two may have a confrontation or interaction based on whether the distance between the historical movement data of a user and the historical movement data of other users in the same scene reaches a certain threshold range, thereby deleting this section of movement data from the entire data. Of course, such a judgment process may mistakenly delete some non-confrontational or interactive data, but since the amount of data itself is relatively sufficient, deleting some non-confrontational data will not have much impact on the training set. Afterwards, since the user will more or less stay, hide, or even make some unnecessary movements (such as turning in circles in place, etc.) according to his or her own operating habits when carrying out activities in the entire virtual scene. Then, a time window can be set to randomly intercept the historical mobile data with a set duration to determine whether the user's operation (movement distance and / or movement mode) in the intercepted data meets the preset movement rules. If so, the corresponding movement data can be deleted from the historical mobile data. The preset movement rules are the corresponding judgment criteria such as staying and hiding set in advance, which can be specifically set according to the specific application scenario.

[0073] In an optional exemplary embodiment, the generation of a training set includes: selecting a continuous motion trajectory of the historical movement data, determining the starting point and the end point of the motion trajectory; starting from the starting point, selecting an intermediate point on the motion trajectory according to a set distance; using the starting point and the end point as inputs of the path midpoint prediction model, using the intermediate point as output of the path midpoint prediction model, and using the input and output of the path midpoint prediction model as a training set. In another optional exemplary embodiment, the input of the path midpoint prediction model is a feature, and the output of the path midpoint prediction model is a label.

[0074] In this embodiment, in order to accurately generate a training set, after completing the screening of historical mobile data and other tasks, the input and output of each data required for the training set can be obtained; wherein, the screening can be to clean the historical mobile data. The input is the starting point and end point of each motion track in the historical mobile data. Since the historical mobile data is cleaned, the originally complete and continuous motion track may be divided into multiple segments, so that a historical mobile data may contain multiple motion tracks. The output is an intermediate point on the motion track with a set distance from the starting point. In a specific application scenario, the set distance is adapted to the distance from the intermediate point finally generated by the model to the starting point. Afterwards, in order to maximize the value, after completing the selection of the first intermediate point, this intermediate point can continue to be used as the new starting point as input, and another intermediate point can be determined as output. Thus, multiple training data are generated on a motion track. That is, in an optional exemplary embodiment, after selecting an intermediate point on the motion track according to the set distance, it also includes: deleting the track from the starting point to the intermediate point on the motion track, generating a new motion track and adding it to the historical mobile data. In a specific embodiment, for a movement trajectory of a single user from the filtered historical movement data, the starting point and the end point are recorded, and then the point at a certain distance from the starting point is used as the next point of the starting point, and then the point at a certain distance from the next point is used as the next point, and so on, to obtain a series of coordinate points. The previous point and the end point are used as input, i.e., features, and the next point is used as output, i.e., a label, to construct a training set. In an optional exemplary embodiment, the training of the path midpoint prediction model through the training set also includes: obtaining the current state data of the virtual object and the action data that the virtual object can perform; performing reinforcement learning on the virtual object based on the state data, the action data, and the next node predicted by the path midpoint prediction model, and performing the next round of reinforcement learning on the virtual object according to the reward obtained from the reinforcement learning, until the preset convergence conditions are met to obtain a trained virtual object.

[0075] In this embodiment, by obtaining the current state data of the virtual object and the action data that the virtual object can perform, the action that the virtual object needs to perform is output through the action reinforcement learning model.; The state data may include environmental information and the current position information of the virtual object; further, the environmental information may include road information, forest information, house information, etc. in the virtual scene; the action data is all the actions that the virtual character can perform. In some scenes, the virtual character can run, jump, climb, use specific props, etc., while in some virtual scenes, the virtual character can only walk, squat, etc. The action reinforcement learning model here is obtained by corresponding training of a general reinforcement learning model. In a specific application scenario, this embodiment mainly trains a basic reinforcement robot (the robot is an intelligent agent in the concept of reinforcement learning, which is the basic concept of the reinforcement learning model) through reinforcement learning. The construction of the reinforcement robot needs to be designed according to specific scene tasks. For example, in a tactical competitive sandbox game, if you want to train a robot with a pathfinding function, you will use the map scene of the real game as the environment of the agent, some game information used for pathfinding as the state (such as current location information, surrounding environment information, target point information, etc.), and the preset actions (behaviors that the robot is expected to perform, such as jumping, using a hook, running, etc.) as the actions that it can perform. Since the purpose is to reach the destination as much as possible, the design of the reward function (the basic concept of the reinforcement learning model) can be related to reaching the destination, such as giving rewards for reaching the destination within a certain number of steps and penalties for not reaching the destination. Other rewards can also be added, such as adding a penalty function for entering a specific regular area (the basic concept of the reinforcement learning model) to enable the robot to learn not to enter a specific regular area. The purpose of this step is to train a robot so that when a target point is given, it can reach this target point as much as possible through a series of behaviors, and the behavior pattern is not too mechanical. That is, in some optional exemplary embodiments, the calculation of the action reinforcement learning model includes: establishing a reward function and / or penalty function of the action reinforcement learning model according to the scene data, the action data and the path-finding target position to perform the calculation of the action reinforcement learning model.

[0076] In an optional exemplary embodiment, after controlling the virtual object to move toward the pathfinding target position, it also includes: in response to the virtual object reaching the pathfinding target position within a set number of steps, taking the pathfinding target position as the new current position of the virtual object, redetermining the pathfinding target position based on the new current position, and controlling the virtual object to move toward the redetermined pathfinding target position.

[0077] Alternatively, in an optional exemplary embodiment, after controlling the virtual object to move toward the pathfinding target position, it also includes: in response to the virtual object failing to reach the pathfinding target position within a set number of steps, reacquiring the current position of the virtual object, redetermining the pathfinding target position based on the reacquired current position, and controlling the virtual object to move toward the redetermined pathfinding target position.

[0078] In the above two embodiments, when the virtual object moves, there are generally two situations, namely, reaching the pathfinding target position within the set number of steps or not reaching the pathfinding target position. The set number of steps can be set in advance, or calculated according to the specific pathfinding length, etc. In the scenario of reaching the pathfinding target position, the current solution can be continued with the current position as the starting point to calculate the next pathfinding target position. In the scenario of not reaching the pathfinding target position, the current movement is terminated in time, and the next pathfinding target position is recalculated, and the route is replanned to make the movement more humanized. Finally, the virtual object reaches the final destination.

[0079] In an optional exemplary embodiment, the path midpoint prediction model is a fully connected neural network model.

[0080] Among them, the fully connected neural network model is essentially a single switch that connects all inputs and outputs. It has the characteristics of high throughput, high reliability and low latency.

[0081] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a path-finding device for a virtual scene.

[0082] refer to Figure 2 , the virtual scene path-finding device comprises:

[0083] An acquisition module 210, used to acquire a current position and a destination position of a virtual object, wherein the virtual object is an intelligent agent obtained through reinforcement learning;

[0084] A determination module 220, configured to input the current position and the destination position into a path midpoint prediction model to calculate an intermediate node position, and use the obtained intermediate node position as a target in the virtual object reinforcement learning process;

[0085] The control module 230 is used to use the intermediate node position as the next pathfinding target position of the virtual object and control the virtual object to move toward the pathfinding target position.

[0086] For the convenience of description, the above devices are described in terms of functions and modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0087] The device of the above-mentioned embodiment is used to implement the path-finding method of the corresponding virtual scene in the above-mentioned embodiment, and has the beneficial effects of the corresponding path-finding method embodiment of the virtual scene, which will not be repeated here.

[0088] In an optional exemplary embodiment, the determining module 220 is further configured to:

[0089] Determine a virtual scene where the virtual object is currently located, and obtain at least one piece of historical movement data of the virtual object controlled by the user in the virtual scene;

[0090] Generate a training set according to the historical movement data, each data of the training set includes a starting point and an end point corresponding to each movement step, and a label composed of a next point of the starting point, wherein the next point is a point on a path formed by the starting point and the end point;

[0091] The path midpoint prediction model is trained using the training set.

[0092] In an optional exemplary embodiment, the determining module 220 is further configured to:

[0093] Generating a training set according to the historical movement data comprises:

[0094] generating a visual path map according to the historical movement data;

[0095] Cleaning the historical movement data according to the visualized path map;

[0096] A training set is generated according to the cleaned historical movement data.

[0097] In an optional exemplary embodiment, the determining module 220 is further configured to:

[0098] Using different styles to display the movement paths of different players, the movement paths forming the visual path map;

[0099] and, if there is an aggregation between two moving paths of different styles in the visualized path graph, the historical moving data corresponding to the aggregation area of ​​the two moving paths of different styles is deleted from the historical moving data; or / and,

[0100] If the corresponding displacement of a single moving path in the visual path diagram is less than a preset distance, the single moving path is deleted from the historical movement data.

[0101] In an optional exemplary embodiment, the determining module 220 is further configured to:

[0102] Inputting the first data in the training set into the path midpoint prediction model, and outputting the next point predicted by the path midpoint prediction model;

[0103] Comparing the predicted next point with the label in the first piece of data, and determining whether a preset convergence condition is met according to the comparison result, and if not, adjusting the parameters of the path midpoint prediction model based on the comparison result;

[0104] The path midpoint prediction model after parameter adjustment is trained for the next round based on the training set until a preset convergence condition is met, thereby obtaining a trained path midpoint prediction model.

[0105] In an optional exemplary embodiment, the control module 220 is further configured to:

[0106] Acquiring current state data of the virtual object and action data that the virtual object can perform;

[0107] Reinforcement learning is performed on the virtual object based on the scene state data, the action data and the next node predicted by the path midpoint prediction model, and the next round of reinforcement learning is performed on the virtual object according to the reward obtained from the reinforcement learning until the preset convergence conditions are met to obtain a trained virtual object.

[0108] In an optional exemplary embodiment, the control module 230 is further configured to:

[0109] In response to the virtual object reaching the pathfinding target position within a set number of steps, the pathfinding target position is used as the new current position of the virtual object, the pathfinding target position is redetermined based on the new current position, and the virtual object is controlled to move toward the redetermined pathfinding target position.

[0110] In an optional exemplary embodiment, the control module 230 is further configured to:

[0111] In response to the virtual object not reaching the pathfinding target position within a set number of steps, the current position of the virtual object is reacquired, the pathfinding target position is re-determined based on the re-acquired current position, and the virtual object is controlled to move toward the re-determined pathfinding target position.

[0112] In an optional exemplary embodiment, the path midpoint prediction model is a fully connected neural network model.

[0113] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the path-finding method for the virtual scene described in any of the above embodiments is implemented.

[0114] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0115] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0116] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0117] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0118] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0119] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0120] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0121] The electronic device of the above-mentioned embodiment is used to implement the path-finding method of the corresponding virtual scene in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0122] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the path-finding method of the virtual scene as described in any of the above embodiments.

[0123] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0124] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the path-finding method of the virtual scene as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0125] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0126] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0127] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0128] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A path-finding method for a virtual scene, characterized in that: include: Obtaining a current position and a destination position of a virtual object, wherein the virtual object is an intelligent agent obtained through reinforcement learning; Inputting the current position and the destination position into a path midpoint prediction model to calculate an intermediate node position, and using the obtained intermediate node position as a target in the virtual object reinforcement learning process; Using the intermediate node position as the next pathfinding target position of the virtual object, and controlling the virtual object to move toward the pathfinding target position; The training of the path midpoint prediction model includes: Determine a virtual scene where the virtual object is currently located, and obtain at least one piece of historical movement data of the virtual object controlled by the user in the virtual scene; Generate a training set according to the historical movement data, each piece of data in the training set includes a starting point and an end point corresponding to each movement step, and a label consisting of a next point of the starting point, wherein the next point is a point on a path formed by the starting point and the end point; The path midpoint prediction model is trained by the training set; the training includes: Inputting the first data in the training set into the path midpoint prediction model, and outputting the predicted next point through the path midpoint prediction model; Comparing the predicted next point with the label in the first piece of data, and determining whether a preset convergence condition is met according to the comparison result, and if not, adjusting the parameters of the path midpoint prediction model based on the comparison result; Performing a next round of training on the path midpoint prediction model after parameter adjustment based on the training set until a preset convergence condition is met, thereby obtaining a trained path midpoint prediction model; The method further comprises: Acquire the current state data of the virtual object and the action data that the virtual object can perform; wherein the state data includes: environmental information and current position information of the virtual object; Reinforcement learning is performed on the virtual object based on the state data, the action data and the next node predicted by the path midpoint prediction model, and the next round of reinforcement learning is performed on the virtual object according to the reward obtained from the reinforcement learning until a preset convergence condition is met to obtain a trained virtual object.

2. The method according to claim 1, characterized in that Generating a training set according to the historical movement data comprises: generating a visual path map according to the historical movement data; Cleaning the historical movement data according to the visualized path map; A training set is generated according to the cleaned historical movement data.

3. The method according to claim 2, characterized in that The generating a visual path map according to the historical movement data includes: Using different styles to display the movement paths of different players, the movement paths forming the visual path map; Cleaning the historical movement data according to the visualized path graph includes: If there is an aggregation between two moving paths of different styles in the visualized path graph, the historical moving data corresponding to the aggregation area of ​​the two moving paths of different styles are deleted from the historical moving data; or / and, If the corresponding displacement of a single moving path in the visual path diagram is less than a preset distance, the single moving path is deleted from the historical movement data.

4. The method according to claim 1, characterized in that: After controlling the virtual object to move toward the path finding target position, the method further includes: In response to the virtual object reaching the pathfinding target position within a set number of steps, the pathfinding target position is used as the new current position of the virtual object, the pathfinding target position is redetermined based on the new current position, and the virtual object is controlled to move toward the redetermined pathfinding target position.

5. The method according to claim 1, characterized in that: After controlling the virtual object to move toward the path finding target position, the method further includes: In response to the virtual object not reaching the pathfinding target position within a set number of steps, the current position of the virtual object is reacquired, the pathfinding target position is re-determined based on the re-acquired current position, and the virtual object is controlled to move toward the re-determined pathfinding target position.

6. The method according to claim 1, characterized in that The path midpoint prediction model is a fully connected neural network model.

7. A virtual scene path-finding device, characterized in that: include: An acquisition module, used to acquire the current position and the destination position of a virtual object, wherein the virtual object is an intelligent agent obtained through reinforcement learning; A determination module, used for inputting the current position and the destination position into a path midpoint prediction model to calculate an intermediate node position, and using the obtained intermediate node position as a target in the virtual object reinforcement learning process; A control module, configured to use the intermediate node position as the next pathfinding target position of the virtual object, and control the virtual object to move toward the pathfinding target position; The training of the path midpoint prediction model in the determination module includes: Determine a virtual scene where the virtual object is currently located, and obtain at least one piece of historical movement data of the virtual object controlled by the user in the virtual scene; Generate a training set according to the historical movement data, each piece of data in the training set includes a starting point and an end point corresponding to each movement step, and a label consisting of a next point of the starting point, wherein the next point is a point on a path formed by the starting point and the end point; The path midpoint prediction model is trained by the training set; the training includes: Inputting the first data in the training set into the path midpoint prediction model, and outputting the predicted next point through the path midpoint prediction model; Comparing the predicted next point with the label in the first piece of data, and determining whether a preset convergence condition is met according to the comparison result, and if not, adjusting the parameters of the path midpoint prediction model based on the comparison result; Performing a next round of training on the path midpoint prediction model after parameter adjustment based on the training set until a preset convergence condition is met, thereby obtaining a trained path midpoint prediction model; The path-finding device also includes a virtual object training module, including: Acquire the current state data of the virtual object and the action data that the virtual object can perform; wherein the state data includes: environmental information and current position information of the virtual object; Reinforcement learning is performed on the virtual object based on the state data, the action data and the next node predicted by the path midpoint prediction model, and the next round of reinforcement learning is performed on the virtual object according to the reward obtained from the reinforcement learning until a preset convergence condition is met to obtain a trained virtual object.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to implement the method according to any one of claims 1 to 6.

Images