Reinforcement learning-based NPC combat decision-making method and related product

Abstract

Disclosed in the present application are a reinforcement learning-based NPC combat decision-making method and a related product. The method comprises: acquiring the current state information of a target NPC, the current state information of a combat object, the current relative pose information, and the historical combat state information of the target NPC versus the combat object, and processing the current state information of the target NPC, the current state information of the combat object and the current relative pose information, so as to obtain an concatenated feature; by means of a long short-term memory network, obtaining a first processing result output by the long short-term memory network; and, on the basis of the first processing result, contextual information of skill release of the target NPC, a skill set of the target NPC, and bonus items of reinforcement learning, by means of reinforcement learning, performing decision making on a combat strategy that should be taken by the target NPC at the next moment. According to the method provided in the present application, after the information required by reinforcement learning is acquired, NPC combat decisions are made by means of reinforcement learning without needing to establish a complex behavior tree, thereby reducing the labor cost.

Description

An NPC battle decision-making method based on reinforcement learning and related products

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 26, 2023, with application number 2023112509261 and application name “A NPC battle decision-making method based on reinforcement learning and related products”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to an NPC battle decision-making method based on reinforcement learning and related products. Background Art

[0003] With the continuous advancement of technology, people's entertainment options are expanding, and gaming is one of the most popular pastimes. NPCs, short for non-player characters, are a type of character in games that are not controlled by real players. NPCs can provide various in-game services and experiences, enhancing the realism and interactivity of the game. NPCs are crucial in games. They can advance the game's plot, assign quests to the player character, trade with the player character, and even assist the player character in combat.

[0004] Combat NPCs face a wide variety of scenarios and enemies in the game world, fighting a wide variety of enemies in various scenarios. They must make different combat decisions for each enemy in each scenario. For example, facing ten low-level enemies in a plains scenario, the NPC might make Combat Decision A; facing a single elite enemy in a volcano scenario, the NPC might make Combat Decision B. Related technologies often use behavior trees to determine combat NPCs' combat decisions. A behavior tree is a tree with a clearly structured node hierarchy that controls the NPC's combat strategies.

[0005] However, using behavior trees to determine the combat decisions of NPCs in open virtual scenes often requires a lot of manpower costs. Therefore, how to reduce the manpower cost used to determine the combat decisions of NPCs has become a technical problem that needs to be solved urgently in this field.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an NPC battle decision-making method and related products based on reinforcement learning, aiming to solve the problem of consuming a large amount of manpower costs in determining NPC battle decisions in related technologies.

[0008] The first aspect of the present application provides an NPC battle decision-making method based on reinforcement learning, comprising:

[0009] Obtaining current state information of a target NPC in a virtual scene, current state information of the target NPC's opponent, current relative position information between the target NPC and the opponent, and historical battle state information between the target NPC and the opponent; the current state information of the target NPC includes contextual information of the target NPC's skill release and the target NPC's skill set;

[0010] Extracting and splicing features based on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information to obtain spliced ​​features;

[0011] Processing the spliced ​​features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network;

[0012] Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reinforcement learning reward item, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning; wherein, the reward value of the reward item is determined based on weakening the combat capability of the combat object.

[0013] The second aspect of the present application provides an NPC battle decision-making device based on reinforcement learning, comprising:

[0014] An information acquisition module is configured to acquire current state information of a target NPC in a virtual scene, current state information of the target NPC's opponent, current relative position information between the target NPC and the opponent, and historical battle state information between the target NPC and the opponent; the current state information of the target NPC includes contextual information for the target NPC's skill release and the target NPC's skill set;

[0015] A splicing module, configured to extract and splice features based on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information to obtain spliced ​​features;

[0016] a long short-term memory network module, configured to process the spliced ​​features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network;

[0017] A reinforcement learning module is used to make a decision on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reinforcement learning reward item; wherein the reward value of the reward item is determined based on weakening the combat capability of the combat object.

[0018] A third aspect of the present application provides an NPC battle decision-making device based on reinforcement learning, the device comprising a processor and a memory:

[0019] The memory is used to store a computer program and transmit the computer program to the processor;

[0020] The processor is used to execute the steps of the NPC battle decision method based on reinforcement learning provided in the first aspect according to the instructions in the computer program.

[0021] The fourth aspect of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by an NPC battle decision device based on reinforcement learning, it implements the steps of the NPC battle decision method based on reinforcement learning provided in the first aspect.

[0022] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by an NPC battle decision-making device based on reinforcement learning, implements the steps of the NPC battle decision-making method based on reinforcement learning provided in the first aspect.

[0023] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0024] The technical solution of this application uses the current state information of the target NPC in the virtual scene, the current state information of the opponent, and the current relative posture information to extract and splice features to obtain spliced ​​features. The spliced ​​features and historical battle state information are processed by a long short-term memory network to obtain a first processing result output by the long short-term memory network. Then, based on the first processing result, the context information of the target NPC releasing skills, the target NPC's skill set, and the reward items of reinforcement learning, a decision is made on the battle strategy that the target NPC should adopt at the next moment through reinforcement learning. Since the spliced ​​features are obtained using information at a certain moment, the spliced ​​features can well represent the battle state information at a certain moment. The battle state information at this moment and the historical battle state information are used in the long short-term memory network for processing to obtain a first processing result. The first processing result is the result obtained by combining the battle state information at this moment and the battle state information at a certain moment in history. The first processing result combines the current battle state information and the historical battle state information, which helps to more accurately determine the target NPC's battle strategy at the next moment. By using reinforcement learning to decide the combat strategy that the target NPC should adopt at the next moment based on the first processing result and related information, it is very simple and efficient to determine the target NPC's combat decision at the next moment in different scenarios by simply obtaining relevant information, performing certain processing, and setting reward items. Compared with related technologies, there is no need to establish a large number of complex behavior trees, which saves a lot of manpower costs and improves processing efficiency and decision-making accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of a behavior tree for a combat NPC;

[0026] FIG2 is a scenario architecture diagram of an NPC battle decision-making method based on reinforcement learning provided in an embodiment of the present application;

[0027] FIG3 is a flow chart of an NPC battle decision-making method based on reinforcement learning provided in an embodiment of the present application;

[0028] FIG4 is a schematic diagram of an application scenario of an NPC battle decision-making method based on reinforcement learning provided in an embodiment of the present application;

[0029] FIG5 is a schematic diagram of the structure of an NPC battle decision-making method based on reinforcement learning provided in an embodiment of the present application;

[0030] FIG6 is a network structure diagram of an NPC battle decision-making method based on reinforcement learning provided in an embodiment of the present application;

[0031] FIG7 is a schematic diagram of the structure of an NPC battle decision-making device based on reinforcement learning provided in an embodiment of the present application;

[0032] FIG8 is a schematic structural diagram of a server in an embodiment of the present application;

[0033] FIG9 is a schematic structural diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0034] As mentioned above, a behavior tree is a tree with a clear node hierarchy. You can use behavior trees to control a series of combat strategies of combat NPCs.

[0035] Figure 1 is a schematic diagram of a behavior tree for a combat NPC. The situation information includes the battle scene and enemies. The combat NPC's combat style is selected through the style selection node. For example, Style 1 can be an aggressive, damage-for-damage combat style, Style 2 can be a balanced combat style, and Style 3 can be a conservative combat style that prioritizes maintaining the combat NPC's status. After selecting a style, the system passes through the self-state determination node, the opponent's state determination node, and the distance determination node, and then arrives at the strategy selection node. The strategy selection node selects either Strategy 1 or Strategy 2. Strategy 1 contains a wait node, a move node, and a move node. The move node can contain skill nodes that the combat NPC can use, also known as the move selection node. When the skill selected by the move selection node is executed, the execution result is returned to the situation information node.

[0036] The situation information in the behavior tree diagram for the combat NPC shown in Figure 1 refers to a specific situation. In one possible implementation, the situation includes a battle against three low-level enemies in a forest game scene. Therefore, all combat decisions made by this behavior tree are specific to the situation in which the combat NPC fights three low-level enemies in the forest game scene. If, for example, a battle against five fish enemies in a deep-sea game scene occurs, the behavior tree shown in Figure 1 will not work. A behavior tree corresponding to the deep-sea game scene against five fish enemies will need to be created to determine the combat strategy for the combat NPC in this scenario.

[0037] A game can potentially present a vast number of combat scenarios. For each scenario, the relevant technology requires using the scenario information to build a corresponding behavior tree, which is then used to determine the NPC's combat strategy. Different NPCs may have different behavior trees for the same scenario. In one possible implementation, given the same scenario information, NPC1 may only have two styles, while NPC2 may have five. This means that for the same scenario information, the behavior trees for NPC1 and NPC2 are completely different. This results in the need to build a very large number of behavior trees to determine NPC combat strategies, further leading to high labor costs.

[0038] In view of the above problems, this application provides an NPC battle decision-making method and related products based on reinforcement learning, with the aim of determining the NPC's battle decision using lower labor costs.

[0039] The execution subject of the NPC battle decision method based on reinforcement learning provided in the embodiment of the present application can be a computer device, and the computer device can include a terminal device or a server. For example, information in the virtual scene is obtained on the terminal device. As an example, the terminal device can specifically include but is not limited to mobile phones, desktop computers, tablet computers, laptop computers, handheld computers, vehicle-mounted terminals and aircraft, etc. The execution subject of the NPC battle decision method based on reinforcement learning provided in the embodiment of the present application can also be a server, that is, the spliced ​​features and historical battle status information can be processed on the server through a long short-term memory network to obtain the first processing result output by the long short-term memory network. The NPC battle decision method based on reinforcement learning provided in the embodiment of the present application can also be executed collaboratively by a terminal device and a server. Therefore, in the embodiment of the present application, there is no limitation on the implementation subject of the technical solution of the present application.

[0040] Figure 2 illustrates an exemplary scenario architecture for a reinforcement learning-based NPC combat decision-making method. The diagram includes servers and various terminal devices. The server shown in Figure 2 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. Furthermore, the server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0041] FIG3 is a flow chart of a reinforcement learning-based NPC battle decision-making method provided in an embodiment of the present application. The method can be executed by a computer device. In this embodiment, the computer device is a terminal device. The reinforcement learning-based NPC battle decision-making method shown in FIG3 includes:

[0042] S301: Acquire the current state information of the target NPC in the virtual scene, the current state information of the target NPC's opponent, the current relative position information of the target NPC and the opponent, and the historical battle state information of the target NPC and the opponent.

[0043] The virtual scene may be a game scene. In a possible implementation, the virtual scene may be an open virtual scene, where an open virtual scene refers to a virtual scene with a high degree of freedom, such as an open game world.

[0044] The terminal device obtains the current state information of the target NPC, the current state information of the target NPC's opponent, the current relative position information of the target NPC and the opponent, and the historical battle state information between the target NPC and the opponent. The target NPC is an NPC in the virtual scene. In one possible implementation, the target NPC can be a combat NPC in an open virtual scene.

[0045] The current state information of the target NPC obtained by the terminal device includes the contextual information for the target NPC's skill release and the target NPC's skill set. In one possible implementation, the target NPC is currently releasing a skill A that requires a 20-second release time. At the moment the terminal device obtains the target NPC's current state information, the target NPC has already released skill A for 5 seconds. The target NPC's current state information includes information indicating that skill A has been released for 5 seconds. The target NPC can use ten skills: skills A, B, ..., and J. The terminal device obtains the target NPC's skill set consisting of these ten skills. The contextual information for releasing skills may include any residual effects of skills that have not yet been completed. In one possible implementation, the target NPC releases skill A at a certain moment, and the effect of skill A is to continuously damage the target NPC's opponent for 10 seconds. Five seconds later, the contextual information for releasing skills may include information such as skill A has been causing damage to the target NPC's opponent for 5 seconds and will continue to cause damage to the target NPC's opponent for an additional 5 seconds.

[0046] In one possible implementation, the target NPC's coordinates in the virtual scene at the current moment are (100, 300, 200), with the front of the target NPC facing the positive x-axis. This information can serve as the target NPC's current state information. The target NPC's health is 300, its attack power is 50, its defense power is 30, and its critical strike rate is 50%. This information can also serve as the target NPC's current state information. The target NPC's current state information can also include the target NPC's teammates' current combat ability attributes, movement information, teammate skill context information, and teammate skill sets.

[0047] The information about the target NPC's current state is the current state of the target NPC's opponent. In one possible implementation, the coordinates of the target NPC's opponent are (105, 302, 201), and the front of the target NPC's opponent faces the negative half of the y-axis. This information can be used as the current state of the target NPC's opponent. The target NPC's opponent has 5000 health points, 100 attack points, 10 defense points, and a 10% critical strike rate. This information can also be used as the current state of the target NPC's opponent.

[0048] The current relative position information between the target NPC and the opponent is used to represent the relative posture and relative position of the target NPC and the opponent. In one possible implementation, the coordinates of the target NPC are (100, 300, 200), and the coordinates of the opponent the target NPC is fighting are (105, 302, 201). The coordinates of the target NPC and the coordinates of the opponent the target NPC is fighting can be used to determine the current relative position of the target NPC and the opponent. The front of the target NPC is facing the opponent, and the front of the opponent is facing the negative half-axis of the y-axis. The relative orientation information between the front of the target NPC and the opponent can be used as the relative posture.

[0049] The historical battle status information between the target NPC and the opponent refers to the historical battle status between the target NPC and the opponent at a certain moment before the current moment.

[0050] S302: Extract and combine features based on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information to obtain combined features.

[0051] The terminal device extracts features of the current state information of the target NPC, the current state information of the opponent, and the current relative posture information, and obtains features that can be recognized by the neural network after feature extraction. Different feature types will have different processing methods during feature extraction. In one possible implementation, the current state information of the target NPC is discrete category data, which can be processed using encoding methods such as One Hot Encoding; the current relative posture information is a continuous feature, which can be processed using encoding methods such as fully connected network coding. The terminal device obtains multiple features after feature extraction, and splices the multiple features to obtain spliced ​​features. In one possible implementation, the spliced ​​features can be long vectors.

[0052] S303: Processing the spliced ​​features and historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network.

[0053] The long short-term memory (LSTM) network combines short-term memory with long-term memory through gate control. The LSTM network can better handle the problems of long-term information storage and short-term input jumping.

[0054] The concatenated features represent the characteristics of a specific moment, while the historical battle status information represents the battle status information before that moment. The long short-term memory network processes the concatenated features and historical battle status information to produce the first processing result output by the long short-term memory network. The LSTM network allows for a better fusion of the concatenated features and historical battle status information from a temporal perspective, establishing a temporal relationship between the concatenated features and historical battle status information. This information is reflected in the first processing result, providing a basis for subsequent strategic decisions.

[0055] S304: Based on the first processing result, the context information of the target NPC releasing the skill, the target NPC's skill set and the reward value of the reinforcement learning reward item, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning.

[0056] Reinforcement learning can capture the actions performed by a target NPC in a virtual environment. Each action is associated with a reward, representing the desired outcome for the target NPC. Generally, weakening the opponent's combat capability is a desired outcome for the target NPC, so weakening the opponent's combat capability is typically used as a reward. Based on this weakening of the opponent's combat capability, the reward value of the reward can be determined. Generally speaking, the greater the weakening of the opponent's combat capability, the greater the reward value of the corresponding reward.

[0057] In one possible implementation, the reward item can be the percentage of health reduction of the opponent. For example, the target NPC performing action 1 can reduce the health of the opponent by 2%, and the target NPC performing action 2 can reduce the health of the opponent by 5%. Action 1 and action 2 will both have corresponding reward values. Because action 2 reduces the health of the opponent by a larger percentage, the reward value of action 2 will be higher than the reward value of action 1.

[0058] In another possible implementation, the reward can take into account both the opponent's health reduction percentage and the target NPC's health reduction percentage. For example, if the target NPC performs action 3, the opponent's health will be reduced by 2% and the target NPC's own health will be reduced by 5%. If the target NPC performs action 4, the opponent's health will be reduced by 5% and the target NPC's own health will be reduced by 20%. Both actions 3 and 4 have corresponding reward values. Although action 4 reduces the opponent's health by a larger percentage, action 4 also reduces the target NPC's health by a larger percentage. Therefore, considering both actions 3 and 4, the reward value for action 3 will generally be higher than that for action 4.

[0059] In another possible implementation, the reward item may be a combination of the opponent's health reduction percentage and the target NPC's health reduction percentage, with certain weights set for the opponent's health reduction percentage and the target NPC's health reduction percentage to comprehensively obtain the final reward value.

[0060] Besides directly reducing the target's health by a certain percentage, weakening the target's combat capability can also involve reducing its current status information. In one possible implementation, the target NPC performing Action 5 could reduce the target's defense by 50%, while Action 6 could reduce the target's health by 3%. Both Actions 5 and 6 have corresponding reward values. Taking both into account, the reward for Action 5 is generally higher than that for Action 6.

[0061] In one possible implementation, the reward may also take into account the time it takes to defeat the opponent. For example, if the target NPC takes thirty seconds to defeat the opponent by performing action 7, and only twenty seconds to defeat the opponent by performing action 8, both actions 7 and 8 will have corresponding reward values. Taking both actions 7 and 8 into consideration, the reward value for action 8 will be higher than the reward value for action 7.

[0062] In another possible implementation, the reward may also take into account the distance between the target NPC and the opponent. When the target NPC performs action 9, the distance between them remains relatively large. When the target NPC performs action 10, the distance between them is very close. Both actions 9 and 10 will have corresponding reward values. Taking both actions 9 and 10 into consideration, the reward for action 10 will be higher than that for action 9.

[0063] Reinforcement learning rewards are determined based on actual needs. Generally, they include multiple indicators, each with a corresponding weight. In one possible implementation, the reward value can be obtained by weighted summing the individual reward values ​​of the multiple reward items. The terminal device uses reinforcement learning to determine the target NPC's next combat strategy.

[0064] In one possible implementation, a terminal device extracts features from the contextual information of a target NPC's skill release and the target NPC's skill set to obtain multiple skill features for the target NPC. The skill set includes the name, effect tag, cooldown, release distance, and current availability of each skill. For example, the target NPC has two skills, Skill A and Skill B. Skill A's skill name is "Recover," its effect tag is "Recover 10% of the target's health," its cooldown is 30 seconds, its release distance is 1, and its current availability is "Available." Skill B's skill name is "Fireball," its effect tag is "Reduce the opponent's health by 20 points," its cooldown is 2 seconds, its release distance is 20, and its current availability is "Unavailable." The skill set includes the aforementioned multiple features corresponding to Skill A and the aforementioned multiple features corresponding to Skill B. The features of Skill A and Skill B are concatenated to obtain a concatenated skill feature result. Based on the first processing result, the concatenated skill feature result, and the reinforcement learning reward, a reinforcement learning method is used to determine the target NPC's next combat strategy.

[0065] It can be seen that, considering that the first processing result reflects the overall characteristics of the target NPC and the combat object in the global time sequence, and the target NPC skills are one of the key factors in making strategic decisions for the target NPC, in order to increase the proportion of features related to the target NPC skills in subsequent decisions, the context information of the target NPC's skill release and the target NPC's skill set are first feature spliced, and then the obtained skill feature splicing results and the first processing results are used for reinforcement learning, so that in reinforcement learning, clearer and more weighted target NPC skill information can be obtained, and ultimately the accuracy of the combat strategy that should be adopted by the target NPC at the next moment through reinforcement learning is improved.

[0066] Based on the first processing result, the skill feature splicing result, and the reward value of the reward item obtained by the combat strategy adopted at the previous moment, the first probability distribution, second probability distribution, and third probability distribution of the target NPC at the next moment are predicted. The first probability distribution is the probability distribution of the skills to be released. In one possible implementation, the target NPC has two skills, Skill A and Skill B. The description of Skill A and Skill B in this paragraph is consistent with the description in the previous paragraph. Since Skill A is currently available and Skill B is currently unavailable, the probability of releasing Skill A in the first probability distribution is 100%, and the probability of releasing Skill B is 0%. In another possible implementation, in addition to Skills A and B, the target NPC possesses Skill C. Skill C has the skill name "Freeze", the effect tag "Reducing the Movement Speed ​​of the Combatant to 0", a cooldown of 20 seconds, a release distance of 20, and current availability of "Available". The terminal device can predict the probability distribution of the target NPC releasing the three skills at the next moment based on the first processing result and the skill feature splicing result.

[0067] The second probability distribution is the probability distribution for the direction of movement, and the third probability distribution is the probability distribution for the distance of movement. In one possible implementation, the target NPC possesses skill D, the release distance of skill D is 5, and the distance between the target NPC and the opponent shown in the first processing result is 10. In order to release skill D, the target NPC needs to reduce the distance between the target NPC and the opponent by moving. The target NPC can move toward the opponent in a straight line or in an arc. The second probability distribution is the probability distribution for the direction of movement. The target NPC can move toward the opponent in a straight line, the target NPC can move 8 points toward the opponent in a straight line, or the target NPC can move 5 points toward the opponent in a straight line. The third probability distribution is the probability distribution for the distance of movement. In another possible implementation, the target NPC possesses skill E, the effect of skill E is to restore 50 health points to the target NPC. When the target NPC releases skill E, it does not need to move or turn at all. In this case, the target NPC can be directly controlled to release skill E without any movement or turning. In another possible implementation, the target NPC masters skill F. The effect of skill F is to attack hostile characters within a range of 10 centered on the target NPC. The target NPC does not need to move or turn when releasing skill F. At this time, the target NPC can be directly controlled to release skill F without any movement or turning.

[0068] In one possible implementation, the second probability distribution includes a macro-movement direction probability distribution and a micro-movement direction probability distribution. Micro-movement refers to a small movement in the virtual environment, for example, the target NPC moves three pixels to the left. Macro-movement refers to a large-scale movement of the target NPC in the virtual environment, for example, the target NPC moves from coordinates (0,0,0) to coordinates (0,0,1000) in the virtual environment.

[0069] The third probability distribution includes a macro-movement distance probability distribution and a micro-movement distance probability distribution. The target macro-movement distance is determined based on the highest value in the macro-movement distance probability distribution, and the target micro-movement distance is determined based on the highest value in the micro-movement distance probability distribution. In one possible implementation, the macro-movement distance can be the distance from one coordinate in an open virtual scene to another, and the micro-movement distance can be the height of a leg lift, the amplitude of a hand swing, etc.

[0070] The target skill is determined based on the highest value in the first probability distribution, the target direction is determined based on the highest value in the second probability distribution, and the target distance is determined based on the highest value in the third probability distribution. In one possible implementation, skill C is the highest value in the first probability distribution, moving in a straight line toward the opponent is the highest value in the second probability distribution, and moving 5 is the highest value in the third probability distribution. Based on this information, a simulation can be performed in which the target NPC releases the target skill, moves in the target direction, and moves the target distance at the next moment. The target NPC releases skill C at the second moment and moves 5 along a straight line toward the opponent. When the target NPC releases skill C at the next moment and moves 5 along a straight line toward the opponent, a reward value is obtained. Based on the action at the second moment, the action performed by the target NPC at the third moment after the second moment can be obtained, and the reward value corresponding to the action performed at the third moment can be obtained. The target is to obtain the highest cumulative value of the reward items obtained, and the target is used to determine the combat strategy that the target NPC should adopt at the next moment based on the cumulative value of the reward items.

[0071] The method provided in the present application processes the spliced ​​features and historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network. The method provided in the present application not only takes into account the features of the current moment, but also takes into account the historical battle status information, and can obtain more reasonable target NPC actions when using the first processing result for reinforcement learning. The present application makes decisions on the battle strategy that the target NPC should adopt at the next moment by means of reinforcement learning. As long as the information corresponding to a certain moment is used as the input of reinforcement learning, the battle strategy that the target NPC should adopt at the next moment can be obtained. Compared with related technologies, there is no need to establish a large number of complex behavior trees, which saves a lot of manpower costs while reducing the time required to determine the target NPC's battle strategy.

[0072] In one possible implementation, the target NPC's current state information also includes the target NPC's current combat capability attribute information and movement information. The combat capability attribute information can include the target NPC's current health, attack power, defense power, critical hit rate, and movement speed, or it can be a score evaluating the target NPC's combat capability attributes. For example, the target NPC's current combat capability attribute information can include 200 health, 10 attack power, 30 defense power, 10% critical hit rate, and 50 movement speed. The target NPC's current combat capability attribute information can also be a combat capability score derived by comprehensively evaluating the above information. For example, one point of health is scored as 5, one point of attack power is scored as 10, one point of defense is scored as 3, one point of critical hit rate is scored as 15, and one point of movement speed is scored as 5. Score 200 health, 10 attack power, 30 defense power, 10% critical hit rate, and 50 movement speed, resulting in a combat capability score of 1590. 1590 can be used as the target NPC's current combat capability attribute information. Movement information is information related to the target NPC's movement within the virtual scene.

[0073] The current state information of the opponent, including its current combat capability attributes and movement information, is similar to the current state information of the target NPC, and will not be elaborated on here. The opponent's current state information also includes contextual information about the target NPC's skill release. For example, if the opponent possesses skill X, which takes ten seconds to release, and at a certain point the opponent has already released skill X for three seconds, this information will serve as contextual information for the target NPC's skill release.

[0074] The current relative position information includes the relative position and relative orientation of the combat object and the target NPC.

[0075] The terminal device extracts features of the current state information of the target NPC, the current state information of the combat object, and the current relative posture information, respectively, to obtain the current state features of the target NPC, the current state features of the combat object, and the current relative posture features of the target NPC and the combat object. The terminal device splices multiple features contained in the current state features of the target NPC to obtain a first splicing result; splices multiple features contained in the current state features of the combat object to obtain a second splicing result; and splices multiple features contained in the current relative posture features to obtain a third splicing result. The splicing can be performed in a preset feature order. For example, when the terminal device splices multiple features contained in the current state features of the target NPC, the preset feature order stipulates that the splicing is performed in the order of health features, attack power features, defense power features, critical hit rate features, movement speed features, and movement information features. The order of features in the final first splicing result remains consistent with the preset feature order.

[0076] The terminal device uses a first deep neural network, a second deep neural network, and a third deep neural network to extract features from the first splicing result, the second splicing result, and the third splicing result, respectively, to obtain a first extraction result, a second extraction result, and a third extraction result. In one possible implementation, the first deep neural network, the second deep neural network, and the third deep neural network can use a DNN (Deep-Learning Neural Network) neural network.

[0077] The terminal device splices the first extraction result, the second extraction result, and the third extraction result to obtain a spliced ​​feature.

[0078] In the method provided by this application, feature extraction and splicing are performed on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information of the opponent. This application takes into account the difference between the target NPC and the opponent, and does not extract and splice the current state information of the target NPC and the current state information of the opponent together, but processes them separately. By processing the current state information of the target NPC and the current state information of the opponent separately, the difference between the target NPC and the opponent can be taken into account to a greater extent, and ultimately improve the accuracy of the battle strategy that should be adopted by the target NPC at the next moment through reinforcement learning.

[0079] In one possible implementation, the terminal device can also obtain the global environmental information of the target NPC in the virtual scene. The global environmental information includes time information, the number of combatants, and environmental warning perception information. Among them, the environmental warning perception information includes the warning range before the combatant releases the skill and the lasting effect after the combatant releases the skill. For example, the combatant releases the skill of spitting fire at a certain moment. The impact range of the spitting fire is a fan-shaped range facing the front of the combatant. This fan-shaped range can be used as a warning range before the combatant releases the skill. After the combatant releases the fire, a flame that lasts for five seconds will be formed within the fan-shaped range. The flame that lasts for five seconds formed within the fan-shaped range can be used as a lasting effect after the combatant releases the skill.

[0080] The terminal device can splice the global environment information with the first feature extraction result, the second feature extraction result, and the third feature extraction result to obtain spliced ​​features.

[0081] The method provided in this application takes into account the impact of global environmental information on the target NPC's combat decision-making. It combines the acquired global environmental information with the first feature extraction result corresponding to the target NPC's current state information, the second feature extraction result corresponding to the combat object's current state information, and the third feature extraction result corresponding to the current relative posture information. This allows for more accurate combat decisions of the target NPC during reinforcement learning.

[0082] In one possible implementation, the terminal device can obtain the latest reward value of the reward item obtained after the target NPC releases the target skill at the next moment and moves in the target direction and the target distance. The target skill can be an attack skill. The target NPC moves 20 meters eastward at the next moment and releases skill A to attack the opponent. The latest reward value is obtained after the target NPC moves 20 meters eastward and releases skill A to attack the opponent. In another possible implementation, the terminal device can obtain the latest reward value of the reward item obtained after the target NPC is controlled to release the target skill at the next moment. The target skill can be a healing skill. The target NPC releases the target skill for itself at the next moment and obtains the latest reward value after releasing it.

[0083] The terminal device can make a decision on the combat strategy that the target NPC should adopt at the target moment through reinforcement learning based on the latest reward value, the latest status information of the target NPC in the virtual scene, the latest status information of the target NPC's opponent, the latest relative posture information between the target NPC and the opponent, and the latest historical combat status information between the target NPC and the opponent. The target moment is the moment after the next moment.

[0084] In one possible implementation, if the target NPC moves 20 meters eastward and then uses skill A to attack the opponent at the next moment, it will receive a new reward value of 20. After the target NPC moves 20 meters eastward and then uses skill A to attack the opponent, the relative position information between the target NPC and the opponent, as well as the status information of the target NPC's opponent, will change. The terminal device can use reinforcement learning to decide the combat strategy that the target NPC should adopt at the target moment after the next moment.

[0085] In one possible implementation, the target NPC releases Skill A at the next moment, moving 5 meters in a straight line toward the opponent. After performing this action, the target NPC receives a reward of 30. However, because this action places the target NPC within a flaming fan, the reward is reduced by 10. At the second moment, the reward is used as a criterion for evaluating the target NPC's actions. Based on the action at the second moment, the target NPC's actions performed at the third moment after the second moment can be determined, along with the reward value corresponding to the action performed at the third moment. The cumulative reward values ​​obtained at the second and third moments are used to determine the target NPC's next combat strategy.

[0086] If it is determined that the target NPC will maximize the accumulated reward value if it releases the target skill, moves in the target direction, and moves the target distance at the next moment, then the target NPC will be determined to release the target skill, move in the target direction, and move the target distance at the next moment. In one possible implementation, the target NPC will release skill A at the second moment and move 5 points in a straight line toward the opponent, which maximizes the accumulated reward value of the target NPC. In this case, releasing skill A and moving 5 points in a straight line toward the opponent will be used as the target NPC's combat strategy.

[0087] If it is determined that the target NPC cannot maximize the continuous accumulation of the reward item if it releases the target skill, moves in the target direction, and moves the target distance at the next moment, simulation is used to determine the skill, movement direction, and movement distance that maximize the reward item, and these are used as the skill, movement direction, and movement distance that the target NPC should release at the next moment. In another possible implementation, if the action of moving 5 points along a straight line toward the opponent does not maximize the continuous accumulation of the target NPC's reward item, simulation is used to determine the second action, which is to release skill B and remain in place; and the third action, which is to release skill C and move 6 points along a straight line in the opposite direction of the opponent. Action 3 can maximize the reward item, and the terminal device will use the release of skill C and the movement of 6 points along a straight line in the opposite direction of the opponent as the target NPC's combat strategy.

[0088] By taking the reward value at each moment when the target NPC releases a skill as the latest reward value, the reinforcement learning is instructed to continue making combat strategy decisions at the next moment based on this value. In this way, the target NPC can be continuously controlled to interact with the combat object through reinforcement learning, ensuring the continuity of control.

[0089] FIG4 is a schematic diagram of an application scenario of an NPC battle decision method based on reinforcement learning provided by an embodiment of the present application. Here, the game client exemplarily selects a smartphone. Of course, in addition to smartphones, it can also be a desktop computer, tablet computer, laptop computer, PDA, and car terminal. The game client sends information to the server. The game client sends the current state information of the target NPC, the current state information of the target NPC's opponent, the current relative position information of the target NPC and the opponent, and the historical battle state information of the target NPC and the opponent. The current state information of the target NPC includes the context information of the target NPC's skill release and the skill set of the target NPC. Multiple servers obtain the target NPC's battle decision through reinforcement learning and return the battle decision to the game client.

[0090] Figure 5 is a structural diagram of an NPC battle decision-making method based on reinforcement learning provided in an embodiment of the present application. The Agent in Figure 5 is the target NPC, State represents the current state information of the target NPC, and the current state information of the target NPC is passed through the DNN neural network to obtain policyπθ(s, a), parameterθ is the parameter of the DNN, and policyπθ(s, a) represents the action of the Agent. The action is output to the environment Environment through Take action, the reward value Reward is determined according to the impact of the action on the environment, and the partial state Observe state in the environment Environment is returned to the Agent. The Agent outputs the next action based on the returned reward value Reward and the partial state Observe state.

[0091] Figure 6 is a network diagram of a reinforcement learning-based NPC battle decision-making method provided by this application. In Figure 6, the env_info in the bottom row represents global environment information, relative_info represents the current relative position information of the target NPC and the combatant, enemy_info represents the current state information of the target NPC's combatant, self_info represents the current state information of the target NPC, and lstm_info represents the historical battle state information of the target NPC and the combatant.

[0092] The information in the target NPC's current state information, self_info, is passed through the feature extraction layer, self_layer, to obtain features corresponding to the target NPC's current state information. These features are then concatenated through the concatenation layer, self_concat, to obtain a long vector corresponding to the target NPC's current state information. This long vector is then input into the first deep neural network (DNN) to obtain a first extraction result. The contextual information of the target NPC's skill release and the target NPC's skill set within the target NPC's current state information are then input into the feature extraction layer, self_layer, to obtain skill-related features. These skill-related features are then input into the concatenation layer, skill_concat, and concatenated into a long vector through the concatenation layer, skill_concat.

[0093] If the current status information of the target NPC includes the current status information of the NPC as a teammate, a pooling layer can be added to ensure that the dimension of the output features is consistent.

[0094] The information in the current state information enemy_info of the battle object is passed through the feature extraction layer enemy_layer to obtain the features corresponding to the current state information of the battle object, the features corresponding to the current state information of the battle object are input into the splicing layer enemy_concat to obtain the long vector corresponding to the current state information of the battle object, and the long vector corresponding to the current state information of the battle object is input into the second deep neural network DNN to obtain the second extraction result.

[0095] The information in the current relative posture information relative_info of the target NPC and the combat object is passed through the feature extraction layer relative_layer to obtain the features corresponding to the current relative posture information, the features corresponding to the current relative posture information are input into the concatenation layer relative_concat to obtain the long vector corresponding to the relative posture information, and the long vector corresponding to the relative posture information is input into the third deep neural network DNN to obtain the third extraction result.

[0096] In the connection layer concat, the global environment information, the first extraction result, the second extraction result and the third extraction result are spliced ​​together to obtain the spliced ​​features.

[0097] The concatenated features and historical battle status information (lstm_info) are processed using a long short-term memory (LSTM) network, yielding a first processing result. Based on this first processing result, the contextual information surrounding the target NPC's skill release, features corresponding to the target NPC's skill set, and reinforcement learning rewards, multiple output heads are generated through reinforcement learning. These output heads may include a skill release output head, a direction determination output head, and a distance determination output head. Each output head has a corresponding π(·|s) representing the probability distribution corresponding to that output head. For example, the skill release output head corresponds to the first probability distribution of the target NPC's skill release, the direction determination output head corresponds to the second probability distribution of the target NPC's movement direction, and the distance determination output head corresponds to the third probability distribution of the target NPC's movement distance. Each output head also has a corresponding function value v(s), which is used to evaluate the quality of the output head's output action. This function value v(s) is related to the reward. The output header of the skill release will use the skill-related long vector output by the splicing layer skill_concat. The splicing layer skill_concat ensures that the output header of the skill release corresponds to the target NPC's skills.

[0098] The network structure used in this application obtains the output of the action through the input of the state, and can represent the mapping relationship from the state to the action. Depending on the input state, this application can also use different network structures for processing. In one possible implementation, the input state feature is a continuous feature or a discrete feature, and a DNN neural network can be used. In another possible implementation, the input feature is a two-dimensional plane grid structure, such as a single-channel image for displaying the warning range before the opponent releases a skill. In this case, the single-channel image can be discretized into a grid and processed by the CNN network.

[0099] Based on the NPC battle decision method based on reinforcement learning provided in the previous embodiment, this application also provides an NPC battle decision device 700 based on reinforcement learning. The following is an explanation in conjunction with Figure 7. Figure 7 is a schematic diagram of the structure of an NPC battle decision device based on reinforcement learning provided in an embodiment of this application. The NPC battle decision device based on reinforcement learning shown in Figure 7 includes:

[0100] Information acquisition module 701 is used to obtain current state information of a target NPC in a virtual scene, current state information of the target NPC's opponent, current relative position information between the target NPC and the opponent, and historical battle state information between the target NPC and the opponent; the current state information of the target NPC includes context information for the target NPC's skill release and the target NPC's skill set;

[0101] A splicing module 702 is configured to extract and splice features based on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information to obtain spliced ​​features;

[0102] A long short-term memory network module 703 is configured to process the concatenated features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network;

[0103] The reinforcement learning module 704 is used to make a decision on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reinforcement learning reward item; wherein the reward value of the reward item is determined based on weakening the combat capability of the combat object.

[0104] In one possible implementation, the current state information of the target NPC also includes the current combat capability attribute information and movement information of the target NPC; the current state information of the combat object includes the current combat capability attribute information and movement information of the combat object and context information of the skill released by the combat object target NPC; the current relative posture information includes the relative position and relative orientation of the combat object and the target NPC;

[0105] The splicing module is specifically used for:

[0106] Performing feature extraction on the current state information of the target NPC, the current state information of the combat object, and the current relative posture information, respectively, to obtain current state features of the target NPC, current state features of the combat object, and current relative posture features of the target NPC and the combat object;

[0107] Splicing multiple features included in the current state feature of the target NPC to obtain a first splicing result;

[0108] splicing a plurality of features included in the current state feature of the combat object to obtain a second splicing result;

[0109] Splicing multiple features included in the current relative posture feature to obtain a third splicing result;

[0110] Using a first deep neural network, a second deep neural network, and a third deep neural network to respectively extract features from the first splicing result, the second splicing result, and the third splicing result to obtain a first extraction result, a second extraction result, and a third extraction result;

[0111] The first extraction result, the second extraction result, and the third extraction result are spliced ​​together to obtain spliced ​​features.

[0112] In a possible implementation, the NPC battle decision-making device based on reinforcement learning further includes:

[0113] A global environment information acquisition module is used to obtain global environment information of the target NPC in the virtual scene, wherein the global environment information includes time information, the number of combatants, and environmental warning perception information; wherein the environmental warning perception information includes the warning range before the combatant releases a skill and the lasting effect after the combatant releases the skill;

[0114] The splicing module is specifically used for:

[0115] The global environment information, the first feature extraction result, the second feature extraction result, and the third feature extraction result are spliced ​​together to obtain spliced ​​features.

[0116] In one possible implementation, the reinforcement learning module is specifically used to:

[0117] Performing feature extraction on the context information of the target NPC's skill release and the target NPC's skill set to obtain multiple skill features of the target NPC; the skill set includes the name, effect label, cooldown time, release distance, and current availability of each skill of the target NPC;

[0118] Splicing the multiple skill features to obtain a skill feature splicing result;

[0119] Based on the first processing result, the skill feature splicing result and the reward value of the reinforcement learning reward item, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning.

[0120] In one possible implementation, the reinforcement learning module is specifically configured to:

[0121] Based on the first processing result, the skill feature splicing result, and the reward value of the reward item obtained by the battle strategy adopted at the previous moment, predict a first probability distribution, a second probability distribution, and a third probability distribution of the target NPC at the next moment, wherein the first probability distribution is a probability distribution of the skill released, the second probability distribution is a probability distribution of the direction of movement, and the third probability distribution is a probability distribution of the distance of movement;

[0122] determining a target skill based on the highest value in the first probability distribution, determining a target direction based on the highest value in the second probability distribution, and determining a target distance based on the highest value in the third probability distribution;

[0123] If the target skill is an attack skill with a movement effect, control the target NPC to release the target skill at the next moment, move in the target direction, and move the target distance;

[0124] If the target skill is an attack skill without a movement effect or the target skill is a healing skill, the target NPC is controlled to release the target skill at the next moment.

[0125] The device further includes a target time battle strategy determination module:

[0126] Obtaining the latest reward value of the reward item obtained after controlling the target NPC to release the target skill at the next moment, move in the target direction and move the target distance, or obtaining the latest reward value of the reward item obtained after controlling the target NPC to release the target skill at the next moment;

[0127] Based on the latest reward value, the latest status information of the target NPC in the virtual scene, the latest status information of the target NPC's opponent, the latest relative posture information between the target NPC and the opponent, and the latest historical battle status information between the target NPC and the opponent, a decision is made on the battle strategy that the target NPC should adopt at the target moment through reinforcement learning; the target moment is the moment after the next moment.

[0128] In one possible implementation, the reinforcement learning includes multiple reward items, and the reinforcement learning module is further used to decide on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the weighted sum of the reward values ​​of the multiple reward items. Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reinforcement learning reward item, a decision on the combat strategy that the target NPC should adopt at the next moment is made through reinforcement learning.

[0129] In one possible implementation, the current status information of the target NPC also includes the current combat capability attribute information, movement information, context information of the teammates' skill releases, and the teammates' skill sets of teammates who belong to the same camp as the target NPC.

[0130] An embodiment of the present application provides a reinforcement learning-based NPC battle decision-making device, which can be a server. Figure 8 is a schematic diagram of a server structure provided by an embodiment of the present application. The server 900 can vary significantly due to different configurations or performance, and can include one or more central processing units (CPUs) 922 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage medium 930 can be either transient or persistent storage. The program stored in the storage medium 930 can include one or more modules (not shown), each of which can include a series of instruction operations on the server. Furthermore, the CPU 922 can be configured to communicate with the storage medium 930 to execute the series of instruction operations in the storage medium 930 on the server 900.

[0131] The server 900 may also include one or more power supplies 926 , one or more wired or wireless network interfaces 950 , one or more input and output interfaces 958 , and / or one or more operating systems 941 .

[0132] The CPU 922 is configured to execute the following steps:

[0133] Obtaining current state information of a target NPC in a virtual scene, current state information of the target NPC's opponent, current relative position information between the target NPC and the opponent, and historical battle state information between the target NPC and the opponent; the current state information of the target NPC includes contextual information of the target NPC's skill release and the target NPC's skill set;

[0134] Extracting and splicing features based on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information to obtain spliced ​​features;

[0135] Processing the spliced ​​features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network;

[0136] Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward item of reinforcement learning, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning; wherein the reward value of the reward item is determined based on weakening the combat capability of the combat object.

[0137] The present application also provides another NPC battle decision-making device based on reinforcement learning. The NPC battle decision-making device based on reinforcement learning can be a terminal device. As shown in Figure 9, for the sake of convenience, only the parts related to the present application embodiment are shown. For specific technical details not disclosed, please refer to the method section of the present application embodiment. Take the terminal device as a mobile phone as an example:

[0138] FIG9 is a block diagram illustrating a portion of the structure of a mobile phone according to an embodiment of the present application. Referring to FIG9 , the mobile phone includes components such as a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will appreciate that the mobile phone structure illustrated in FIG9 does not limit the mobile phone and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0139] The following is a detailed introduction to the various components of the mobile phone in conjunction with Figure 9:

[0140] RF circuitry 1010 can be used to receive and transmit signals during information transmission or calls. Specifically, it receives downlink information from the base station and transmits it to processor 1080 for processing. It also transmits uplink data to the base station. Typically, RF circuitry 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, and more. RF circuitry 1010 can also communicate with the network and other devices via wireless communications. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile Communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (English full name: General Packet Radio Service, GPRS), Code Division Multiple Access (English full name: Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (English full name: Wideband Code Division Multiple Access, English abbreviation: WCDMA), Long Term Evolution (English full name: Long Term Evolution, English abbreviation: LTE), email, Short Messaging Service (English full name: Short Messaging Service, SMS), etc.

[0141] The memory 1020 can be used to store software programs and modules. The processor 1080 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0142] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 1031) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1080. It can also receive commands sent by the processor 1080 and execute them. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0143] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 9, the touch panel 1031 and the display panel 1041 are used as two independent components to implement the input and output functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to implement the input and output functions of the mobile phone.

[0144] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0145] Audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the phone. Audio circuit 1060 converts received audio data into electrical signals and transmits them to speaker 1061, which then converts them into sound signals for output. Microphone 1062, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1060 and converted into audio data. The audio data is then processed by processor 1080 and transmitted to, for example, another phone via RF circuit 1010, or stored in memory 1020 for further processing.

[0146] WiFi is a short-range wireless transmission technology. A mobile phone uses WiFi module 1070 to help users send and receive emails, browse the web, and access streaming media, providing wireless broadband Internet access. Although FIG9 shows WiFi module 1070, it is understood that it is not a required component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0147] Processor 1080 is the control center of the phone, connecting all parts of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 1020 and accessing data stored in memory 1020, it executes various phone functions and processes data, thereby collecting data and information about the phone as a whole. Optionally, processor 1080 may include one or more processing units; preferably, processor 1080 may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1080.

[0148] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing charging, discharging, and power consumption through the power management system.

[0149] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0150] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the following functions:

[0151] Obtaining current state information of a target NPC in a virtual scene, current state information of the target NPC's opponent, current relative position information between the target NPC and the opponent, and historical battle state information between the target NPC and the opponent; the current state information of the target NPC includes contextual information of the target NPC's skill release and the target NPC's skill set;

[0152] Extracting and splicing features based on the current state information of the target NPC, the current state information of the opponent, and the current relative posture information to obtain spliced ​​features;

[0153] Processing the spliced ​​features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network;

[0154] Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward item of reinforcement learning, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning; wherein the reward value of the reward item is determined based on weakening the combat capability of the combat object.

[0155] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program. When the computer program runs on an NPC battle decision device based on reinforcement learning, the NPC battle decision device based on reinforcement learning is used to execute any one of the implementation methods of the NPC battle decision method based on reinforcement learning described in the aforementioned embodiments.

[0156] An embodiment of the present application also provides a computer program product including a computer program, which, when run on an NPC battle decision device based on reinforcement learning, enables the NPC battle decision device based on reinforcement learning to execute any one of the implementation methods of the NPC battle decision method based on reinforcement learning described in the aforementioned embodiments.

[0157] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the system is merely a logical function division. In actual implementation, there may be other division methods, such as multiple systems can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0159] The systems described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment as needed.

[0160] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store computer programs.

[0162] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A NPC battle decision-making method based on reinforcement learning, the method being executed by a computer device, the method comprising: Acquire current status information of a target NPC in a virtual scene, current status information of a combatant of the target NPC, current relative position information between the target NPC and the combatant, and historical combat status information between the target NPC and the combatant; the current status information of the target NPC includes context information of a skill released by the target NPC and a skill set of the target NPC; Extract and splice features based on the current state information of the target NPC, the current state information of the opponent, and the current relative position information to obtain spliced ​​features; Processing the spliced ​​features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network; Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reinforcement learning reward item, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning; wherein the reward value of the reward item is determined based on weakening the combat capability of the combat object.

2. According to the method of claim 1, the extracting and splicing of features based on the current state information of the target NPC, the current state information of the fighting object and the current relative posture information to obtain the spliced ​​features comprises: Extract features of the current state information of the target NPC, the current state information of the combat object, and the current relative posture information, respectively, to obtain current state features of the target NPC, current state features of the combat object, and current relative posture features of the target NPC and the combat object; Splicing multiple features included in the current state feature of the target NPC to obtain a first splicing result; splicing a plurality of features included in the current state feature of the combat object to obtain a second splicing result; Splicing multiple features included in the current relative posture feature to obtain a third splicing result; Using a first deep neural network, a second deep neural network, and a third deep neural network to respectively extract features from the first splicing result, the second splicing result, and the third splicing result to obtain a first extraction result, a second extraction result, and a third extraction result; The first extraction result, the second extraction result and the third extraction result are spliced ​​to obtain spliced ​​features.

3. The method according to claim 2, further comprising: Acquire global environmental information of the target NPC in the virtual scene, wherein the global environmental information includes time information, the number of combat objects, and environmental warning perception information; wherein the environmental warning perception information includes a warning range before the combat object releases a skill and a lasting effect after the combat object releases a skill; The step of splicing the first extraction result, the second extraction result, and the third extraction result to obtain a spliced ​​feature includes: The global environment information, the first feature extraction result, the second feature extraction result and the third feature extraction result are spliced ​​to obtain spliced ​​features.

4. The method according to claim 1, wherein the step of deciding the combat strategy to be adopted by the target NPC at the next moment by means of reinforcement learning based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reinforcement learning reward item comprises: Extracting features from the context information of the target NPC releasing the skill and the skill set of the target NPC to obtain multiple skill features of the target NPC; The skill set includes the name, effect label, cooldown time, release distance and current availability of each skill of the target NPC; Splicing the multiple skill features to obtain a skill feature splicing result; Based on the first processing result, the skill feature splicing result and the reward value of the reinforcement learning reward item, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning.

5. The method according to claim 4, wherein the step of deciding the combat strategy to be adopted by the target NPC at the next moment by means of reinforcement learning based on the first processing result, the skill feature splicing result and the reward value of the reward item of reinforcement learning comprises: Based on the first processing result, the skill feature splicing result and the reward value of the reward item obtained by the battle strategy adopted at the previous moment, predict the first probability distribution, the second probability distribution and the third probability distribution of the target NPC at the next moment, wherein the first probability distribution is the probability distribution of the skill released, the second probability distribution is the probability distribution of the direction of movement, and the third probability distribution is the probability distribution of the distance of movement; determining a target skill based on the highest value in the first probability distribution, determining a target direction based on the highest value in the second probability distribution, and determining a target distance based on the highest value in the third probability distribution; If the target skill is an attack skill with a movement effect, control the target NPC to release the target skill at the next moment, move in the target direction, and move the target distance; If the target skill is an attack skill without a movement effect or the target skill is a healing skill, the target NPC is controlled to release the target skill at the next moment.

6. The method according to claim 5, further comprising: Obtain the latest reward value of the reward item obtained after controlling the target NPC to release the target skill at the next moment, move in the target direction and move the target distance, or obtain the latest reward value of the reward item obtained after controlling the target NPC to release the target skill at the next moment; Based on the latest reward value, the latest status information of the target NPC in the virtual scene, the latest status information of the target NPC's opponent, the latest relative posture information between the target NPC and the opponent, and the latest historical battle status information between the target NPC and the opponent, a decision is made on the battle strategy that the target NPC should adopt at the target moment through reinforcement learning; the target moment is the moment after the next moment.

7. The method according to any one of claims 1 to 6, wherein the reinforcement learning includes a plurality of reward items, and the step of making a decision on the combat strategy to be adopted by the target NPC at the next moment by means of reinforcement learning based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reward item of the reinforcement learning comprises: Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the weighted sum of the reward values ​​of the multiple reward items, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning. Based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reward item of reinforcement learning, a decision is made on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning.

8. According to the method of claim 1, the current status information of the target NPC also includes the current combat capability attribute information, movement information, context information of the teammates releasing skills and the skill set of the teammates who belong to the same camp as the target NPC.

9. An NPC battle decision-making device based on reinforcement learning, comprising: An information acquisition module is used to acquire current state information of a target NPC in a virtual scene, current state information of an opponent of the target NPC, current relative position information between the target NPC and the opponent, and historical battle state information between the target NPC and the opponent; the current state information of the target NPC includes context information of a skill released by the target NPC and a skill set of the target NPC; A splicing module, used for extracting and splicing features based on the current state information of the target NPC, the current state information of the fighting object and the current relative posture information to obtain spliced ​​features; A long short-term memory network module, used for processing the spliced ​​features and the historical battle status information through a long short-term memory network to obtain a first processing result output by the long short-term memory network; A reinforcement learning module is used to make a decision on the combat strategy that the target NPC should adopt at the next moment through reinforcement learning based on the first processing result, the context information of the target NPC releasing the skill, the skill set of the target NPC and the reward value of the reinforcement learning reward item; wherein the reward value of the reward item is determined based on weakening the combat capability of the combat object.

10. An NPC battle decision-making device based on reinforcement learning, the device comprising a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute the steps of the NPC battle decision method based on reinforcement learning according to any one of claims 1 to 8 according to the instructions in the computer program.

11. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program, when executed by an NPC battle decision device based on reinforcement learning, implements the steps of the NPC battle decision method based on reinforcement learning as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, which, when executed by an NPC battle decision device based on reinforcement learning, implements the steps of the NPC battle decision method based on reinforcement learning as described in any one of claims 1 to 8.

Images