Intelligent agent control method and system based on deep reinforcement learning

Through deep reinforcement learning methods, multi-layer perception mechanisms and attention mechanisms are used to generate feature vectors, and combined with policy networks and reward functions to optimize action decisions, the decision efficiency and effectiveness of traditional agent control methods in complex environments is solved, and intelligence is improved The body's decision-making ability and gaming experience.

CN118267709BActive Publication Date: 2024-11-19北京华清飞扬网络股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410275750.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-12
Publication Date
2024-11-19
Estimated Expiration
2044-03-12

AI Technical Summary

Technical Problem

Traditional rules-based agent control methods are difficult to cope with complex and dynamically changing multi-agent environments, resulting in poor decision-making efficiency and effectiveness.

Method used

Using a method based on deep reinforcement learning, the state information of agents and hostile agents is obtained through multi-layer perception mechanisms, feature vectors are generated, and target enemies are determined using attention mechanisms, and combined with policy networks, value function networks and reward functions to optimize action decisions .

Benefits of technology

It improves the decision-making ability and gaming experience of the agent, and enhances the intelligent decision-making effect in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118267709B_ABST
    Figure CN118267709B_ABST
Patent Text Reader

Abstract

The present invention discloses an intelligent agent control method and system based on deep reinforcement learning, comprising: first, obtaining the state information of the current intelligent agent and the hostile intelligent agent through a multi-layer perception mechanism, and generating a feature vector. Then, the attention mechanism is used to determine the target enemy. At the same time, the encoding features of the game map where the intelligent agent is located are obtained and extracted. These features are input into the policy network to obtain action commands. Then, the predicted evaluation results and actual reward results of the action commands are obtained respectively using the preset value function network and reward function. Finally, the policy network is optimized according to these two results to make better action decisions. Such a design combines deep learning and reinforcement learning, improves the decision-making ability of the intelligent agent, and thus enhances the gaming experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent agent control method and system based on deep reinforcement learning. Background Art

[0002] In computer games, especially multi-agent esports games, improving the decision-making efficiency and effectiveness of agents (i.e., artificial intelligence characters) is a crucial research topic. Traditional rule-based approaches, overly reliant on pre-set rules and judgment logic, are unable to cope with complex and dynamically changing game environments. Consequently, research on agent control using artificial intelligence techniques such as deep learning is gaining increasing attention. Summary of the Invention

[0003] The purpose of the present invention is to provide an intelligent agent control method and system based on deep reinforcement learning.

[0004] In a first aspect, an embodiment of the present invention provides an agent control method based on deep reinforcement learning, comprising:

[0005] Obtaining current state information of a current agent and hostile state information of multiple hostile agents, and inputting the current state information of the current agent and the hostile state information of the multiple hostile agents into an entity scalar encoder constructed based on a multi-layer perception mechanism to obtain an entity feature vector of the current agent and hostile feature vectors corresponding to each of the multiple hostile agents;

[0006] Inputting the entity feature vector and the plurality of hostile feature vectors into a target enemy selection unit constructed based on an attention mechanism, and determining a target hostile agent from the plurality of hostile agents;

[0007] Obtaining the game map where the current agent is located, and extracting the map coding features of the current agent;

[0008] Inputting the entity feature vector, the target hostile feature vector corresponding to the target hostile agent, and the map encoding feature into a strategy network to obtain an action command for instructing the current agent whether to perform a shooting operation on the target hostile agent;

[0009] Obtaining a prediction evaluation result of the action command based on a preset value function network;

[0010] Obtaining an actual reward result of the action command based on a preset reward function;

[0011] The policy network is optimized and adjusted according to the predicted evaluation result and the actual reward result, so as to make an action decision of the current agent using the optimized and adjusted policy network.

[0012] In a second aspect, an embodiment of the present invention provides a server system, including a server, wherein the server is configured to execute the method described in the first aspect.

[0013] Compared with the existing technology, the beneficial effects provided by the present invention include: using the agent control method and system based on deep reinforcement learning disclosed by the present invention, the state information of the current agent and the hostile agent is obtained through a multi-layer perception mechanism, and a feature vector is generated. The target enemy is then determined using the attention mechanism. At the same time, the encoding features of the game map where the agent is located are obtained and extracted. These features are input into the policy network to obtain action commands. Then, the preset value function network and reward function are used to obtain the predicted evaluation results and actual reward results of the action commands respectively. Finally, the policy network is optimized based on these two results to make better action decisions. This design combines deep learning and reinforcement learning, improves the decision-making ability of the agent, and thus enhances the gaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly describes the drawings required for use in the embodiments. It should be understood that the following drawings illustrate only certain embodiments of the present invention and should not be construed as limiting the scope of the present invention. Those skilled in the art can, without inventive effort, derive other relevant drawings from these drawings.

[0015] Figure 1 A schematic diagram of the steps of the intelligent agent control method based on deep reinforcement learning provided by an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0018] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] In order to solve the technical problems in the above background technology, Figure 1 This is a flow chart of the intelligent agent control method based on deep reinforcement learning provided in an embodiment of the present disclosure. The intelligent agent control method based on deep reinforcement learning is introduced in detail below.

[0020] Step S201: obtaining current state information of the current agent and hostile state information of multiple hostile agents, and inputting the current state information of the current agent and the hostile state information of the multiple hostile agents into an entity scalar encoder constructed based on a multi-layer perception mechanism to obtain an entity feature vector of the current agent and hostile feature vectors corresponding to each of the multiple hostile agents;

[0021] Step S202: inputting the entity feature vector and the plurality of enemy feature vectors into a target enemy selection unit constructed based on an attention mechanism, and determining a target enemy agent from the plurality of enemy agents;

[0022] Step S203, obtaining the game map where the current agent is located, and extracting the map coding features of the current agent;

[0023] Step S204: Input the entity feature vector, the target hostile feature vector corresponding to the target hostile agent, and the map encoding feature into a strategy network to obtain an action command for instructing the current agent whether to perform a shooting operation on the target hostile agent;

[0024] Step S205, obtaining a prediction evaluation result of the action command based on a preset value function network;

[0025] Step S206, obtaining the actual reward result of the action command based on a preset reward function;

[0026] Step S207: Optimize and adjust the policy network according to the predicted evaluation result and the actual reward result, so as to use the optimized and adjusted policy network to formulate the action decision of the current intelligent agent.

[0027] In an exemplary embodiment of the present invention, the server first obtains the current state information of the current agent (e.g., a friendly game character or robot), including its location, health, and equipment status. Simultaneously, the server also obtains enemy state information for multiple enemy agents (enemy characters or robots), such as their location, health, and behavior patterns. In a 10-player shooting game, the server obtains information such as the location (coordinates X1, Y1), health (100%), and equipment status (including rifles and grenades) of a friendly agent, "Warrior A." Simultaneously, the server monitors various state information for nine enemy agents. The server inputs the obtained state information for the current agent and the states of multiple enemy agents into an entity scalar encoder built on a multi-layer perception mechanism. This encoder converts this state information into entity feature vectors and enemy feature vectors. The server inputs the state information of "Warrior A," including its location, health, and equipment, as well as the state information of the nine enemy agents, into the entity scalar encoder, which outputs the entity feature vector for "Warrior A" and the nine enemy feature vectors. The server inputs the entity feature vector and multiple enemy feature vectors into a target enemy selection unit built using an attention mechanism. This unit analyzes these feature vectors and determines, from among multiple enemy agents, the target enemy agent that the current agent should prioritize attacking. By analyzing Warrior A's entity feature vector and nine enemy feature vectors, the target enemy selection unit determines that enemy agent Sniper B (located at X2, Y2, with 80% health and a sniper rifle) poses the greatest threat to Warrior A and therefore selects Sniper B as the target enemy agent. The server obtains information about the game map the current agent is currently on and extracts map-encoded features relevant to the current agent, such as obstacle locations and traversable areas. Warrior A's game map contains multiple buildings, obstacles, and traversable paths. The server analyzes this information and extracts map-encoded features relevant to Warrior A's current location and actions. The server inputs the entity feature vector, target enemy feature vector, and map-encoded features into a policy network. Based on this input information, the policy network outputs an action command, instructing the current agent whether to execute a shooting operation against the target hostile agent. Based on the entity feature vector of "Warrior A", the target hostile feature vector of "Sniper B", and the map encoding features, the policy network outputs an action command, instructing "Warrior A" to move to a more advantageous position before shooting at "Sniper B". The server uses a pre-set value function network to predict and evaluate the action command, obtaining a predicted evaluation result. At the same time, the server also calculates an actual reward result for the action command based on the actual game situation (such as whether the target is hit, the amount of damage caused, etc.). The value function network predicts that the action of "Warrior A" moving to a new position and shooting "Sniper B" will receive a higher reward.In reality, "Warrior A" successfully moved to a new position and dealt damage to "Sniper B." The server calculated an actual reward for this action based on the damage and the game rules. The server then optimized the policy network based on the predicted evaluation and actual reward results to make more accurate action decisions in future games. The server compared the predicted evaluation and actual reward results and discovered a discrepancy. Consequently, the server adjusted the policy network's parameters to more accurately predict the optimal action in similar situations in future games.

[0028] In order to more clearly describe the implementation scheme of the present invention, a more detailed description is given below.

[0029] In the embodiments of the present invention, the current agent, for example, refers to the intelligent entity being watched or controlled. It can be a character, robot, or other type of intelligent agent in a game. In a multiplayer online shooting game, the current agent can be an AI-controlled character on the player's side or the player's enemy side, such as a game character named "Warrior A." Current state information refers to a data set containing the various attributes and conditions of the current agent at a given moment. For "Warrior A," this current state information may include its position coordinates (X1, Y1), health points (e.g., 100%), equipment status (e.g., rifle held, ammunition count), movement speed, etc. A hostile agent refers to an intelligent entity in a hostile relationship with the current agent, typically an opponent or enemy in the game. In a shooting game, a hostile agent can be an enemy AI-controlled character, such as "Sniper B" or other AI-controlled enemies. Hostile state information refers to a data set containing the various attributes and conditions of the hostile agent at a given moment. This information is crucial for the current agent because it determines how the current agent should respond to the enemy. For "Sniper B," enemy state information might include their location coordinates (X2, Y2), health (e.g., 80%), weapon (e.g., sniper rifle), and behavior patterns (e.g., hiding, attacking, etc.). A multi-layer perception mechanism typically refers to a neural network structure in deep learning. It comprises multiple layers of neurons and is capable of learning and extracting complex features from input data. In an entity scalar encoder, this multi-layer perception mechanism can be a neural network consisting of multiple fully connected layers. It receives state information from the current agent and the enemy agent as input, processes this information through neurons in each layer, and ultimately outputs an entity feature vector and an enemy feature vector. The entity scalar encoder is a neural network model that converts the raw state information of the current agent and the enemy agent into a feature representation, or feature vector, that is easier to process and analyze. The entity scalar encoder receives state information such as the location and health of "Warrior A" and state information from enemy agents such as "Sniper B" as input, and outputs an entity feature vector for "Warrior A" and multiple enemy feature vectors. These feature vectors capture the key features of the original state information and can be used for subsequent tasks such as target enemy selection and action decision-making. The entity feature vector is a vector representation of the current agent's state information, while the enemy feature vector is a vector representation of the enemy agent's state information. These vectors are converted using an entity scalar encoder. They contain the key features of the original state information and can be used as input for machine learning models. For "Warrior A", the entity feature vector may be a multidimensional vector containing information such as their position, health points, and equipment status; for "Sniper B", the enemy feature vector is also a multidimensional vector containing information such as their position, health points, and weapons held.These vectors play a key role in the subsequent target enemy selection and action decision-making process.

[0030] Furthermore, in embodiments of the present invention, an attention mechanism is a technique that enables a model to focus its attention on relevant information when processing information. In deep learning, attention mechanisms are typically implemented by calculating weights between different information elements, enabling the model to dynamically focus on the information most important for the current task. In the target enemy selection task, the attention mechanism can help the model calculate a weight distribution based on the state of the current agent and the state information of the hostile agents, indicating which hostile agents the current agent should pay more attention to. The target enemy selection unit is a neural network module based on the attention mechanism. Its function is to select one or more hostile agents from multiple hostile agents as the primary attack targets of the current agent based on the input entity feature vector and multiple hostile feature vectors. In this unit, the attention mechanism calculates the attention weight of the current agent (e.g., "Soldier A") on each hostile agent (e.g., "Sniper B," "Infantry C," and "Tank D"). Then, based on these weights, the hostile agent with the highest weight is selected as the target hostile agent. If "Sniper B" poses the greatest threat to "Warrior A," then "Sniper B" may be selected as the target hostile agent. A target hostile agent is one that, after processing by the target enemy selection unit, is determined to be the enemy that the current agent should prioritize attention or attack in the current situation. In a battle, "Warrior A" may face multiple enemies, but based on its own state and the state of the enemies, the target enemy selection unit calculates "Sniper B" as the enemy that "Warrior A" should pay the most attention to, and therefore "Sniper B" is the target hostile agent. Through this process, the server can leverage the attention mechanism and the target enemy selection unit to dynamically determine the current agent's primary attack target in complex game environments, enabling more intelligent and effective decision-making.

[0031] Furthermore, in embodiments of the present invention, a game map refers to a two-dimensional or three-dimensional data structure representing the layout and environment of the game world. It contains information such as the location, attributes, and relationships of all interactive objects in the game. In a strategy game, the game map might be a two-dimensional planar map showing geographical features such as mountains, rivers, cities, and roads. In a first-person shooter game, the game map might be a three-dimensional space containing complex elements such as buildings, bunkers, and weapon spawn points. Obtaining the game map where the current agent is located refers to extracting information about the portion of the map where the current agent is located, or the entire game map, from the game environment. This information is crucial for the agent's decision-making and actions. For example, in a strategy game, the AI ​​character (i.e., the current agent) is located in a city on the map. "Obtaining the game map where the current agent is located" then means obtaining map data for that city and its surrounding area, including information such as the city's topography, resource distribution, and enemy forces. Map encoding features convert the information in the game map into a numerical or vector form that can be processed and understood by machine learning models. This conversion is typically achieved through feature engineering or deep learning methods. In a simple case, we can divide the game map into a grid, with each grid cell representing a specific area (such as grassland, mountain, or water). We can then use a two-dimensional matrix to represent the entire map, where each matrix element corresponds to a grid cell and contains information about the area's type (e.g., 0 for grassland, 1 for mountain, etc.). This two-dimensional matrix is ​​a kind of map encoding feature. Extracting the current agent's map encoding features involves extracting from the game map the features relevant to its decisions and actions. This feature information might include the terrain type, enemy distribution, and resource locations surrounding the agent. In the aforementioned strategy game example, if the current agent (AI character) is located in a city and preparing to launch an attack, extracting map encoding features might include determining whether the terrain around the city is conducive to attack (e.g., whether there are mountains providing cover), the distribution of enemy forces (e.g., the location and number of enemy units), and the distribution of resources (e.g., whether there are nearby available resource points). This information is converted into numerical or vector form and used as input for the machine learning model.

[0032] Furthermore, in embodiments of the present invention, the policy network is a trained machine learning model, typically a deep learning network. It receives as input information the current agent's state, the target enemy agent's state, and map encoding features, and outputs an action decision or action command. In a shooting game, the policy network can be a neural network that, based on the input entity feature vector, target enemy feature vector, and map encoding features, determines the action the current agent should take, such as moving to a certain location or shooting at the target enemy agent. The action command is the output of the policy network and indicates the specific action the current agent should perform. In a shooting game, action commands may include moving, shooting, reloading, and using items. After processing the input information, the policy network may output an action command instructing "Soldier A" to shoot "Sniper B." This command is then converted into specific in-game actions, such as adjusting the scope or pulling the trigger. By integrating these different feature information (entity feature vectors, target enemy feature vectors, and map encoding features) and inputting them into the policy network, the server can intelligently generate the optimal action command for the current game state, thereby achieving more intelligent and adaptive game AI behavior.

[0033] Furthermore, in embodiments of the present invention, the value function network is a key component in reinforcement learning, used to estimate the long-term rewards of taking an action in a given state. It is typically a neural network that is trained to learn how to predict the value of a state-action pair, that is, the expected sum of future rewards. This network helps the agent consider long-term consequences when choosing actions. In a shooting game, the value function network might receive as input the current game state (including the agent's current position, health, enemy positions, etc.) and a potential action command (such as "shoot" or "move to a certain location"), and output a predicted value representing the total future reward the agent would receive if the action were executed. This predicted value helps the agent determine which action is most likely to be optimal. The action command is the result of the agent's decision based on the current state and environmental information, indicating the specific action the agent should take. In the context of reinforcement learning, action commands are typically generated by a policy network. In a shooting game, an action command might be "shoot at the enemy's location" or "move behind cover." These commands are generated based on the agent's understanding of the current game state and its goal (such as defeating the enemy). The predicted evaluation result is the value function network's prediction of the potential future reward of a given action command. It reflects the expected long-term reward the agent will receive if it performs that action. In a shooting game, when the agent considers whether to shoot an enemy, the value function network provides a predicted evaluation result, representing the total future reward the agent may receive if the action is executed. This predicted evaluation result helps the agent determine whether shooting is a good choice.

[0034] Furthermore, in embodiments of the present invention, a pre-set reward function is a core concept in reinforcement learning. It defines the reward signal an agent receives from the environment after performing a certain action. This reward signal is the basis for the agent to learn how to optimize its behavior. A pre-set reward function means that it is already defined before the agent begins interacting with the environment, and rewards are calculated based on the agent's actions and the state of the environment. In a shooting game, the reward function might be defined as follows: if the agent shoots and hits an enemy, it receives a positive reward; if the agent is hit by an enemy, it receives a negative reward; and if the agent completes a certain task (such as capturing a stronghold), it receives an additional positive reward. These rules are determined at the beginning of game design and are used to evaluate each of the agent's actions. The actual reward is the reward the agent actually receives from the environment after executing the action command. This reward is calculated based on the pre-set reward function and reflects the impact of the action command on the environment and whether it progresses toward the agent's goal. In a shooting game, if the agent executes the action command "shoot at the enemy's location" and successfully hits the enemy, then according to the pre-set reward function, the agent will receive a positive reward as the actual reward. This positive reward indicates that the agent's actions were effective and progressing toward achieving the game's goal. By combining a pre-set reward function with generated action commands, the server can calculate and obtain the reward results of these commands after they are actually executed. These actual reward results provide feedback to the agent, allowing it to understand which actions are beneficial and which are disadvantageous, and thus adjust its strategy to optimize future behavior.

[0035] In addition, in an embodiment of the present invention, optimization adjustment refers to the process of updating and improving the parameters of the policy network based on the action results (predicted evaluation results and actual reward results) of the agent. The purpose is to enable the policy network to make better action decisions in the future to maximize the cumulative rewards obtained from the environment. In a shooting game, if the agent executes the action decision generated by the policy network and obtains an actual reward result, this result will be compared with the previous predicted evaluation result. If there is a difference (for example, the actual reward is lower than expected), the parameters of the policy network will be adjusted through an optimization algorithm so that more accurate action decisions can be generated in similar situations in the future.

[0036] In an embodiment of the present invention, the target enemy selection unit includes a cascaded Softmax architecture and a GumbelSoftmax Sampling architecture, and the aforementioned step S202 can be implemented through the following example.

[0037] Performing a dot product calculation on the entity feature vector and the plurality of hostile feature vectors to obtain a target matrix constructed by multiplying the plurality of vectors, wherein each element in the target matrix is ​​used to represent the correlation between the current agent and each of the hostile agents;

[0038] Normalizing the target matrix using the Softmax architecture to obtain a probability distribution corresponding to the target matrix, wherein each element in the probability distribution is used to represent the probability of the current agent selecting a discrete action of each of the hostile agents;

[0039] Processing the probability distribution using the Gumbel Softmax Sampling architecture to obtain a new probability distribution;

[0040] In the training phase, the loss function is calculated based on the new probability distribution and back propagation is performed to update the model parameters;

[0041] In the inference phase, sampling is performed according to the new probability distribution to determine the final selected target adversarial agent.

[0042] In an embodiment of the present invention, for example, in a multiplayer online shooting game, the server needs to select a target for the current agent (e.g., friendly soldier A) from multiple hostile agents (e.g., enemy soldiers B, C, and D). The server first extracts the entity feature vector of the current agent (Soldier A), which includes information such as Warrior A's position, health points, and equipment. Simultaneously, the server also extracts the enemy feature vectors of enemy soldiers B, C, and D. Next, the server performs a dot product calculation on Warrior A's entity feature vector and the enemy feature vectors of B, C, and D. Through the dot product calculation, the server obtains a target matrix. Each element in this matrix represents the correlation between Warrior A and enemy soldiers B, C, and D. For example, if Warrior A and Warrior B are very close in position and Warrior B has lower health points, then the correlation between Warrior A and Warrior B is likely to be higher. The server uses the Softmax architecture to normalize the target matrix. The Softmax function converts each element in the target matrix into a probability value between 0 and 1, where the sum of all probability values ​​is 1. After Softmax processing, the server obtains a probability distribution. Each element in this probability distribution represents the discrete action probability of Warrior A selecting enemy Warriors B, C, and D as targets. For example, Warrior A has a probability of 0.6 for selecting Warrior B, a probability of 0.3 for selecting Warrior C, and a probability of 0.1 for selecting Warrior D. To better explore different action options during training and avoid consistently selecting the action with the highest probability (i.e., avoiding the problem of overdeterminism), the server uses a Gumbel Softmax Sampling architecture to process the probability distribution. This method adds some randomness (i.e., Gumbel noise) to the original probability distribution, allowing for a wider range of action space to be explored during training. After Gumbel Softmax Sampling, the server obtains a new probability distribution. This new probability distribution retains the general shape of the original probability distribution but adds some randomness. During the training phase of the game, the server needs to optimize the agent's decision-making ability based on the results of its actions. Specifically, the server needs to calculate a loss function and perform backpropagation to update the model parameters. During training, the server calculates the loss function based on the new probability distribution and the actual action results (e.g., whether Warrior A successfully hits the selected target). This loss function measures the discrepancy between the agent's decision and the actual outcome. The server then performs backpropagation using optimization algorithms such as gradient descent, updating the model's parameters to minimize the loss function. Through multiple iterations of training, the model's parameters are gradually optimized, and the agent's decision-making ability improves. For example, when selecting a target, Soldier A can more accurately determine which enemy fighter is the most threatening or easiest to defeat.During the actual game execution phase (i.e., the inference phase), the server needs to quickly and accurately select a target hostile agent for the current agent. During the inference phase, the server performs sampling based on the new probability distribution. Specifically, the server can determine which hostile agent to select as the target based on the magnitude of each element in the probability distribution. For example, if Warrior A has the highest probability of selecting Warrior B, the server selects Warrior B as Warrior A's target. Through sampling, the server ultimately determines the target hostile agent (e.g., Warrior B) for the current agent (Warrior A). Warrior A can then perform the corresponding action based on this decision (e.g., firing at Warrior B).

[0043] In the embodiment of the present invention, the policy network includes a moving policy sub-network and a shooting policy sub-network. The aforementioned step S204 can be implemented through the following example.

[0044] Obtaining a spatial feature map of the map encoding feature, the spatial feature map including obstacle information of the current agent and position information of the target hostile agent;

[0045] Inputting the spatial feature map into the movement strategy subnetwork to obtain a feature map prediction result of a preset size;

[0046] Normalizing the feature map prediction result to obtain a probability distribution of the movement direction of the current agent;

[0047] Performing element-wise addition feature fusion on the entity feature vector, the target hostile feature vector, and the map coding feature to obtain a first fused feature vector;

[0048] Inputting the first fused feature vector into the shooting strategy subnetwork to obtain an action probability distribution of whether the current agent shoots;

[0049] According to the movement direction probability distribution and the action probability distribution, an action command is obtained for indicating whether the current agent performs a shooting operation on the target hostile agent.

[0050] In an embodiment of the present invention, for example, in a multiplayer online shooting game, a server needs to make action decisions for the current agent (e.g., friendly soldier A) including whether to shoot at a target hostile agent (e.g., enemy soldier B) and how to move to optimize the shooting position or avoid obstacles. The server first obtains the encoded features of the map, which include information such as terrain, obstacles, and cover. The server then generates a spatial feature map based on these encoded features. This spatial feature map not only contains information about obstacles surrounding the current agent but also marks the location of the target hostile agent (enemy soldier B). The server obtains a feature map rich in spatial information, providing a basis for subsequent decision making. The server inputs the spatial feature map into the movement strategy subnetwork. This subnetwork is a deep learning model specifically designed to predict the movement direction of the current agent. The movement strategy subnetwork outputs a feature map prediction result of a preset size. This result contains the probability of the current agent moving in different directions. The server normalizes the feature map prediction result and converts it into a probability distribution. In this way, each movement direction is assigned a probability value between 0 and 1, and the sum of all probability values ​​is 1. The server obtains a probability distribution for the current agent's movement direction. This distribution clearly represents the probability of the current agent moving in each direction. The server then fuses the current agent's entity feature vector, the target enemy agent's feature vector, and the map encoding features using element-wise addition. This fusion method preserves the information of each feature vector while effectively combining them. The server generates a first fused feature vector. This vector integrates information about the current agent, the target enemy agent, and the map environment, providing comprehensive input for subsequent shooting decisions. The server inputs this first fused feature vector into the shooting strategy subnetwork, also a deep learning model specifically designed to predict whether the current agent should shoot. The shooting strategy subnetwork outputs an action probability distribution for the current agent's shooting action. This distribution reflects the probability of the current agent shooting or not shooting. The server formulates an action command for the current agent based on the movement direction probability distribution and the action probability distribution. Specifically, the server selects the movement direction and shooting action with the highest probability as the current agent's action command. The server obtains a clear action command, indicating whether the current agent should shoot at the target enemy agent and how to move to optimize its position or avoid risk. This action command is then sent to the current agent for execution.

[0051] In the embodiment of the present invention, the aforementioned step S205 can be implemented through the following example.

[0052] Obtaining pending state information of the current agent and pending state information of the plurality of hostile agents after the current agent executes the action command;

[0053] Extracting the entity state vector of the pending state information of the current agent and the hostile state vectors corresponding to the pending state information of the plurality of hostile agents;

[0054] Averaging the plurality of hostile state vectors to obtain an average hostile state vector;

[0055] Performing element-wise addition feature fusion on the entity state vector and the average enemy state vector to obtain a second fused feature vector;

[0056] The second fused feature vector is input into a preset value function network to obtain a prediction evaluation result of the action command.

[0057] In an embodiment of the present invention, for example, in a multiplayer online shooting game, a server needs to evaluate the effect of a certain action command executed by the current agent (e.g., friendly soldier A). This action command might be "shoot enemy soldier B and move behind cover C." The server first obtains the pending state information of the current agent after executing the action command. This includes the current agent's new position, health points, ammunition level, etc., as well as the pending state information of all enemy agents (e.g., enemy soldiers B, D, and E), such as their position, health points, and whether they are behind cover. The server now has the latest state information of all relevant agents after executing the action command. From this obtained pending state information, the server extracts the entity state vector of the current agent and the enemy state vectors of each enemy agent. These vectors are numerical representations, facilitating subsequent calculations and processing. The server obtains the entity state vector of the current agent and the enemy state vectors of multiple enemy agents. The server then averages these multiple enemy state vectors. This means that the server calculates the average value of all enemy state vectors in each dimension to obtain an average enemy state vector. This vector represents the average state of all enemy agents. The server obtains an average enemy state vector, which combines the state information of all enemy agents. The server then performs an element-wise fusion of the current agent's entity state vector and the average enemy state vector. This means that the elements in corresponding positions in the two vectors are added together to form a new fused feature vector. The server then obtains a second fused feature vector, which contains both the current agent's state and the average state information of all enemy agents. The server then inputs this second fused feature vector into a pre-set value function network. This value function network is a deep learning model that has been trained to predict the long-term rewards or effects of an action command in a given state. The value function network outputs a predicted evaluation result. This result is a numerical value representing the expected effect or reward of executing the action command in the current state. Based on this result, the server can decide whether to execute the action command or choose a more advantageous action.

[0058] In the embodiment of the present invention, the aforementioned step S206 can be implemented through the following example.

[0059] Based on a pre-set reward function: Obtaining the actual reward result of the action command;

[0060] Among them, the Rewards is the actual reward result, the α i Is the reward type R i The weight of .

[0061] In the embodiment of the present invention, in order to avoid the win-loss rewards after each game (Eposide) being too sparse, intermediate rewards are designed, including the following:

[0062] Kill reward R1: When the agent (current intelligent body) kills the enemy, a fixed reward value is given to encourage the agent to kill the enemy.

[0063] Killing penalty R2: When the agent dies, a fixed negative reward is given to encourage the agent to avoid damage.

[0064] Shooting and Hitting Enemy Reward R3: When the Agent shoots and hits the enemy, a reward is given. The value is proportional to the amount of damage caused, which encourages the Agent to shoot the enemy.

[0065] Injury penalty R4: When the agent is injured, an injury penalty is given, the value of which is proportional to the size of the injury value, to encourage the agent to avoid injury.

[0066] Penalty R5 for shooting without hitting the enemy: When the agent's shooting fails to hit the enemy, it is equivalent to wasting bullet resources. A smaller penalty is given to avoid ineffective shooting.

[0067] Movement Reward R6: Movement Reward is the reward obtained by the Agent based on the target enemy found, which is used to prevent the Agent from moving randomly for a long time.

[0068] The total reward is

[0069] where α i The weights of various rewards determine which rewards are more important to the reinforcement learning model. This can also be used to train agents with different styles. For example, a charging agent will have higher weights for movement rewards and kill rewards, while a flanking agent will have a higher penalty for being killed.

[0070] In the embodiment of the present invention, the policy network is obtained in the following manner.

[0071] Obtaining an initial policy network, and interacting with a preset shooting game environment multiple times based on the initial policy network to obtain a game state trajectory;

[0072] Calculating the discounted rewards and advantage values ​​of the various states included in the game state trajectory according to the game state trajectory in combination with a preset value function network;

[0073] updating the initial policy network based on the discounted reward and the advantage value to obtain an intermediate policy network, and updating the preset value function network based on a mean square loss function;

[0074] Iterative training is performed based on the intermediate strategy network and the updated preset value function network to obtain the trained strategy network.

[0075] In an embodiment of the present invention, for example, when developing a shooting game AI, the server first requires an initial policy network as a starting point. This initial policy network can be a simple neural network model whose structure and parameters have been obtained through some basic training or initialization methods. The server loads or initializes an initial policy network. This network will be used for subsequent interactions and training. The server has an initial policy network that can interact with the environment. To train the policy network, the server needs to let it interact multiple times in the shooting game environment to collect game state trajectory data. This data will be used to evaluate the performance of the policy network and guide network updates. The server uses the initial policy network to interact multiple times in the preset shooting game environment. During each interaction, the policy network outputs an action command (such as shoot, move, etc.) based on the current game state. The game environment then updates its state based on this action command and provides a reward signal. The server records this state, action, and reward data to form a game state trajectory. The server collects a series of game state trajectory data that reflects the performance of the initial policy network in the game. To evaluate the performance of the policy network and guide network updates, the server needs to calculate the discounted reward and advantage value for each state in the game state trajectory. Discounted rewards account for the impact of future rewards on the current state, while advantage measures the difference between actual rewards and expected rewards. The server uses a pre-defined value function network to calculate the value function (i.e., expected reward) for each state. Then, based on the reward signal and value function in the game state trajectory, the server calculates the discounted reward and advantage for each state. Discounted rewards are calculated by accumulating future rewards at a certain discount rate, while advantage is the difference between the actual reward and the value function. The server obtains the discounted reward and advantage for each state and uses this data to update the policy network and value function network. With the discounted reward and advantage, the server can begin updating the initial policy network and value function network. The goal of the update is to enable the policy network to output better action commands to obtain higher rewards and to enable the value function network to more accurately predict the value function for each state. The server uses the discounted reward and advantage to update the parameters of the initial policy network. Specifically, the server can adjust the network parameters using methods such as gradient ascent to increase the probability of actions that result in high rewards. Simultaneously, the server uses the mean squared loss function to update the parameters of the pre-defined value function network to enable it to more accurately predict the value function for each state. This process can be repeated multiple times until the policy network and value function network reach a certain level of performance. After multiple iterations of training, the server obtains a trained policy network and an updated value function network. These networks will be used for decision-making and evaluation of the agent in the game.

[0076] In an embodiment of the present invention, the aforementioned step of calculating the discount reward and advantage value of the game state trajectory based on the game state trajectory in combination with a preset value function network can be implemented through the following example.

[0077] According to the formula: Calculate the discounted reward of the game state trajectory; where R t is the discount reward, V φ (s n ) is the state s estimated by the preset value function network n The value function of , γ is the discount factor;

[0078] According to the formula: A t =R t -V φ (s t ) calculates the advantage value of the game state trajectory; wherein, A t is the advantage value.

[0079] In this embodiment of the present invention, the policy network is trained based on self-play. A shooting game client developed in Unity establishes a persistent connection with a Python server to send and receive game status and control commands. The specific process is as follows:

[0080] Let the Actor Strategy Network π θ Interact with the game environment n times and record the game status trajectory (s t , a t , r t )fort∈{1, 2,...,n-1}.

[0081] The game client will send the current environment state s of the game at time t at a fixed period. t , that is, the player, enemy, map and other states mentioned above; the Actor neural network will be based on the state s t , make action a t And returns to the client. The client takes action a t Control the Agent to move and shoot. After the Agent interacts with the environment, the state of the environment changes to s t+1 , according to the change of state, determine action a t Rewards received t , the server stores the game status track (s t , a t , r t )

[0082] Computing discounted rewards and advantage functions

[0083] After n times of data collection is completed, according to (st , a t , r t ) to calculate the discounted reward R in each state t And the advantage function A t

[0084] Calculation of discount rewards:

[0085]

[0086] Where V φ (s n ) is the state s estimated by the value function critic n The value function of r, because the game agent interacts with the environment n times, there is no n+1th state to determine r n , so the estimated value V of the value function critic φ (s n ) instead. γ is a discount factor, which multiplies the future reward by γ and adds it to the current moment. The purpose is to encourage the model to consider long-term rewards. γ is a number between 0 and 1. When γ = 0, it means that the model only considers the current reward and not the future rewards.

[0087] Calculation of advantage function:

[0088] A t =R t -V φ (s t )

[0089] Where V φ (s t ) is the critic network's response to the input state s t The obtained value function. Then calculate the discounted reward R t With V φ (s t ) difference as the advantage value A t , used to evaluate the Actor's actions relative to the benchmark V φ (s t ). It represents the advantage of a certain action relative to the average action in the current state.

[0090] 3. Update the Actor Network

[0091] The loss function of the model actor is as follows:

[0092]

[0093] Among them A t is the advantage value, r t(θ) is the ratio, which is given by the following formula:

[0094]

[0095] where π θ It is the old model parameter of Actor, that is, the model parameter for interacting with the environment to collect data. θ_new is the model parameter of the new actor, which is updated every epoch. t (θ) represents the probability ratio of actions under the new and old strategies.

[0096] Truncation strategy in formula

[0097] min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)A t )

[0098] r t (θ) is limited to (1-∈, 1+∈), where ∈ is a hyperparameter that indicates the range of truncation. The purpose of truncation is to ensure that the gap between the new and old parameters is not too large. t When it is greater than 0, it means that the value of this action is higher than the average value, and the actor minimizing the loss function will make r t (θ) increases, that is, the probability of the new strategy action will increase. On the contrary, when A t When <0, the probability of the action is reduced. The parameter update uses the Adam optimizer and gradient descent algorithm.

[0099] 4. Update the Critic network:

[0100] The loss function of the model critic is as follows:

[0101]

[0102] Among them, R t is the discounted reward, which is the target reward value calculated based on the reward returned by the environment, V φ (s t ) is the value function estimated by the critic. The entire loss function is the mean square loss, which minimizes R as the training process progresses. t With V φ (s t ) so that the estimated value V of the model critic φ (s t ) is close to R t . This enables the model critic to estimate the value of the current state more accurately.

[0103] After k epochs of training, the old model parameters are updated to the newly trained model parameters. The new model parameters are then used to interact with the game and collect new game data. The training and data collection processes are performed alternately, repeating steps 1-4 until training is complete.

[0104] In the embodiments of the present invention, the following implementation modes are also provided.

[0105] When the current agent executes an action command for a shooting operation against the target hostile agent, obtaining a shooting prop module configured by the current agent for executing the shooting operation;

[0106] Acquire continuous position coordinate information of the shooting prop module at a current time point; the continuous position coordinate information represents position coordinate information of the shooting prop module in a recorded time range and a predicted time range associated with the current time point;

[0107] Acquire current prop interaction information of the shooting prop module at the current time point; the current prop interaction information represents the shooting prop state of the shooting prop module at the current time point;

[0108] Based on the continuous position coordinate information and the current prop interaction information, generating prop operation information of the shooting prop module at the current time point; the prop operation information represents the operation mode of the shooting prop module from the current time point to a subsequent time point of the current time point;

[0109] Based on the prop operation information and the current prop interaction information, the shooting prop state of the shooting prop module at a subsequent time point of the current time point is determined, and the shooting prop state of the shooting prop module indicated by the current prop interaction information is switched to the shooting prop state at a subsequent time point of the current time point.

[0110] In an embodiment of the present invention, for example, in a shooting game, a server needs to process an action command from the current agent (e.g., an AI-controlled character) to execute a shooting operation against a target hostile agent (e.g., an enemy character). When the server receives an action command from the current agent to execute a shooting operation against the target hostile agent, it first obtains the shooting prop module configured for the current agent for executing the shooting operation. This shooting prop module can be a weapon equipped by the AI, such as a pistol or rifle. The server successfully obtains information about the shooting prop module configured for the current agent. To accurately simulate the behavior and effects of the shooting prop module, the server needs to know its location information at different points in time. The server obtains the continuous position coordinate information of the shooting prop module at the current point in time. This information includes the position coordinates for a recorded time range (e.g., the past few seconds) and the position coordinates for a predicted time range (e.g., the next few seconds). This can be achieved through physics engine calculations, interpolation algorithms, or prediction models. The server obtains the continuous position coordinate information of the shooting prop module over a period of time. In addition to the position information, the server also needs to understand the state information of the shooting prop module at the current point in time to correctly handle its interaction with the environment. The server obtains the current prop interaction information of the shooting prop module at the current time point. This information may include status information such as the amount of ammunition in the magazine, the weapon's firing rate, and whether reloading is in progress. The server successfully obtains the state information of the shooting prop module at the current time point. Based on the obtained position coordinate information and prop interaction information, the server needs to generate the operation information of the shooting prop module at the current time point. The server combines the continuous position coordinate information and the current prop interaction information to generate the prop operation information of the shooting prop module at the current time point. This information describes how the shooting prop module should operate from the current time point to a subsequent time point (for example, the next frame or within the next few seconds), such as firing, reloading, switching shooting modes, etc. The server generates the operation mode information of the shooting prop module at the current time point. Based on the generated prop operation information, the server needs to determine the state of the shooting prop module at a subsequent time point and perform the corresponding state switch. The server determines the shooting prop state of the shooting prop module at a subsequent time point after the current time point based on the prop operation information and the current prop interaction information. For example, if the action information indicates firing and ammunition is currently sufficient, the state at a later time might be reduced ammunition, muzzle smoke, etc. The server then switches the shooting prop module from the state indicated by the current prop interaction information to the state at the later time. The server successfully determines the state of the shooting prop module at the later time and executes the corresponding state switch. This makes the shooting operation in the game more realistic and accurate.

[0111] To more clearly describe the solution provided by the embodiments of the present invention, a more detailed description is provided below. In a game, a shooting prop module generally refers to a virtual weapon or equipment used to perform shooting operations. This module includes the weapon's appearance, performance parameters (such as range, rate of fire, damage value, etc.), and the interaction logic with other game elements.

[0112] In an embodiment of the present invention, for example, a shooting action command is a specific instruction issued by the current agent to execute a shooting action. This command may include information such as the target, weapon to be used, force and direction of the shot. In games, a shooting prop module typically refers to the virtual weapon or equipment used to execute the shooting action. This module contains the weapon's appearance, performance parameters (such as range, rate of fire, damage value), and interaction logic with other game elements. In a shooting game, the AK-47 rifle in the hands of an AI character is a shooting prop module. This module not only determines the visual effects of the AI's shooting, but also defines physical properties such as accuracy and recoil. In the context of game programming, "retrieval" generally refers to the process of retrieving specific information or resources from the game's memory, database, or other storage location. This information or resources may include data regarding game state, character attributes, item details, etc. When the server needs to process a shooting action, it must first "retrieve" information about the shooting prop module configured by the current agent. This information includes the weapon type, ammunition status, and whether accessories (such as a scope) are equipped. The server retrieves this information by accessing the game's memory or database. When the server receives a shooting action command from the current agent, it retrieves information about the agent's configured prop module to correctly process and execute the shooting action. This includes understanding the weapon type, state, and interaction with other game elements. The current time point refers to the specific point in time that the system is currently processing or considering during simulation or real-time computation. In game development, this typically refers to the moment when the game engine updates the world state. In a 60-frame-per-second game, every 1 / 60 of a second is a new current time point, at which the game engine updates the state of all objects in the game (including the prop module). Continuous position coordinate information refers to the continuous position of an object (in this case, the prop module) in space over a period of time. This position information is typically expressed in coordinate form, such as (x, y, z) coordinates in three-dimensional space. When the AI ​​fires a gun, the bullet is fired from the muzzle and travels along a specific trajectory. During this process, the bullet's position coordinates are updated at each frame (i.e., each current time point), forming a continuous sequence of position coordinate information. The recorded time range refers to the period of time that has been recorded and processed by the system before the current time point. Within this time range, the position coordinate information of the shooting prop module has been recorded. Suppose we have a system that records the flight trajectory of a bullet in the last 5 seconds. If the current time point is 3 seconds after the bullet is fired, then the recorded time range is the first 3 seconds after the bullet is fired. The predicted time range refers to the period of time after the current time point that the system predicts based on existing information (such as the object's speed and direction).Within this time range, the position coordinates of the shooting prop module are calculated based on the prediction algorithm. Continuing with the above example, if we have a system that predicts the flight trajectory of a bullet in the next 2 seconds, then the prediction time range is 2 seconds after the current time point. The system predicts the position coordinates of the bullet in the next 2 seconds based on the current speed, direction, and other relevant factors of the bullet. When the server processes the shooting prop module, it obtains its continuous position coordinate information at the current time point. This information includes the actual position coordinates of the prop in the past (recorded time range) and the predicted position coordinates in the future (predicted time range). These data are critical to accurately simulate the behavior and effects of the shooting prop module in the game world.

[0113] Furthermore, in embodiments of the present invention, current item interaction information refers to information generated or required by the shooting item module when interacting with the game environment or other objects at the current point in time. This information reflects the item's current state, behavior, and relationship with the surrounding environment. When an AI is aiming with a sniper rifle, current item interaction information may include the position of the scope's crosshairs, the remaining ammunition, the weapon's recoil status, and so on. This information is essential for the server to process shooting operations, as it directly affects the outcome and effectiveness of the shot. The shooting item status refers to the specific state of the shooting item module at the current point in time. This includes the item's activation status, ammunition status, cooldown time, and accessory status (such as whether a silencer or scope is equipped). If the AI's sniper rifle is currently loading ammunition, its shooting item status is "Loading." In this state, the AI ​​cannot perform a shooting operation until reloading is complete. The server obtains current item interaction information to understand the item's status and, based on this information, determines whether the AI ​​can execute a shooting operation and the outcome of the shooting operation. When processing a shooting operation, the server needs to obtain the current item interaction information of the shooting prop module at the current time point. This information reflects the current state of the item and its interaction with the game environment, and is crucial for the server to determine whether the shooting operation is valid and what effect it produces. By obtaining and processing this information, the server can accurately simulate the behavior and effects of the shooting prop module in the game world. Item operation information refers to a set of instructions or data generated based on continuous position coordinate information and current item interaction information. It describes the operation mode or behavior of the shooting prop module from the current time point to a subsequent time point. For example, if an AI is aiming and shooting with a sniper rifle, the item operation information may include the coordinates of the aiming point, the firing force, the firing frequency, and so on. This information is calculated by the game engine based on the AI ​​input and the current state of the item module, and is used to guide the item module's shooting operation at a subsequent time point. A subsequent time point refers to a specific time point or points within a certain range after the current time point. In game processing, it is often necessary to predict or calculate the behavior or state of the item module over a period of time in the future to ensure accurate game logic and rendering. If the current time point is the 10th second in the game, then the subsequent time points may be the 11th second, the 12th second, and so on. At these subsequent time points, the game engine will update the state and behavior of the shooting prop module based on the prop operation information, such as updating the bullet position and determining whether it hits the target. When processing the shooting operation, the server generates prop operation information based on the continuous position coordinate information and the current prop interaction information. This information describes the operation mode or behavior of the shooting prop module from the current time point to the subsequent time point and is an important basis for the server to perform game logic judgment and screen rendering.By accurately calculating and updating this information, the server is able to simulate a realistic and smooth shooting experience.

[0114] In addition, in an embodiment of the present invention, switching the shooting prop state refers to the process of changing the prop module from one state to another according to the game logic and AI operation. This usually involves updating the internal properties and behavior patterns of the prop module. When the AI ​​performs a reload, the prop module will switch from the "shootable" state to the "reloading" state. During this process, the game engine will update the prop module's properties such as the ammunition quantity and reloading progress, and may temporarily disable the shooting function until the reloading is completed. When processing a shooting operation, the server will first determine the shooting prop state of the shooting prop module at a subsequent time point based on the prop operation information and the current prop interaction information. The server will then switch the prop module from the state indicated by the current prop interaction information to the state at a subsequent time point. This process involves updating the prop module properties and adjusting the behavior pattern to ensure the consistency of the game logic and the smoothness of the player experience.

[0115] In an embodiment of the present invention, the continuous position coordinate information is composed of past position coordinate information and expected position coordinate information; wherein, the past position coordinate information is the position coordinate information of the recorded time range associated with the shooting prop module at the current time point; the expected position coordinate information is obtained after analyzing the initial expected position coordinate information and the operation command for the shooting prop module at the current time point, and the initial expected position coordinate information is the position coordinate information of the predicted time range associated with the shooting prop module at the current time point based on the output of the pre-trained position prediction model.

[0116] In an embodiment of the present invention, for example, in a first-person shooter game, a server needs to process a pistol module used by an AI. The current time is the 10th second, and the server has recorded the pistol module's position coordinate information from the 1st to the 10th second. This position coordinate information constitutes the pistol module's past position coordinate information. This information reflects the pistol module's movement trajectory and position changes over the past 10 seconds. In the same shooting game, the server also uses a pre-trained position prediction model to predict the pistol module's position coordinate information over a period of time (e.g., the next 2 seconds). This prediction is based on factors such as the pistol module's movement state, speed, and direction over the past period of time (e.g., the previous 8 seconds). The predicted position coordinate information is called the initial expected position coordinate information. It represents the pistol module's expected position if no new AI operation commands intervene. At the 10th second, the AI ​​simulates pressing the left mouse button to fire and slightly adjusts the pistol's aiming direction. These operation commands are captured and analyzed by the server. The server needs to evaluate the impact of these operation commands on the pistol module's future position. The server analyzed the pistol module and found that the AI's shooting actions caused recoil in the pistol module, altering its original trajectory. Furthermore, adjustments to the aiming direction also affected the pistol module's future position. Therefore, the server adjusted the initial expected position coordinate information based on these factors, generating adjusted expected position coordinate information. This adjusted information more accurately reflects the pistol module's expected position after accounting for the AI's actions. Finally, the server combined the past position coordinate information (the position coordinates from the 1st to the 10th second) with the adjusted expected position coordinate information (the predicted position coordinates from the 11th to the 12th second) to construct the pistol module's continuous position coordinate information. This continuous position coordinate information includes not only the pistol module's past positions but also its expected future positions after accounting for the AI's actions. The server uses this information to accurately simulate the pistol module's movement and shooting effects in the game world.

[0117] In the embodiments of the present invention, the following implementation modes are also provided.

[0118] Obtaining a plurality of historical moments in a recorded time range associated with the current time point;

[0119] Obtaining trajectory coordinate points, direction data, and speed data of the shooting prop module at each of the multiple historical moments;

[0120] The past position coordinate information is generated based on the trajectory coordinate points, direction data and speed data of the shooting prop module at each historical moment.

[0121] In an embodiment of the present invention, for example, a server is processing a sniper rifle module used by an AI in a battlefield shooting simulation game. The current time point is the 20th second in the game. To generate past position coordinate information, the server first needs to obtain multiple historical moments within the recorded time range associated with the 20th second. For example, the server can select every 0.5 seconds as a historical moment. Thus, from the 1st to the 20th second, the server obtains a total of 40 historical moments (1 second, 1.5 seconds, 2 seconds, and so on, up to 20 seconds). For each of these 40 historical moments, the server obtains the sniper rifle module's position coordinates (trajectory coordinate points), direction (e.g., aiming direction), and velocity data when moving or firing. This data fully records the sniper rifle module's position, direction, and velocity changes in the game world. At the 5th second, the sniper rifle module's position coordinates are (100, 200, 300), its direction is northeast, and its velocity is 0 (because the AI ​​is likely aiming and not moving). At the 10th second, the position coordinates change to (110, 210, 310), the heading changes to due east, and the velocity remains at 0. At the 15th second, due to slight movement and aiming adjustments by the AI, the position coordinates change to (115, 215, 315), the heading changes to southeast, and the velocity increases to 2 meters per second. This data continues for all 40 historical moments. The server now has the trajectory coordinates, heading, and velocity data for the sniper module at each historical moment. This data is used to generate past position information. Past position information is a continuous data set that details the position changes of the sniper module over the recorded time range (from the 1st to the 20th second). The server may store this data in memory in some form (such as a data structure or graph) for subsequent use in game logic processing, such as shooting operations, collision detection, and visual rendering. For example, when the AI ​​fires at the 20th second, the server can use this past position information to calculate the bullet's initial flight trajectory and final impact point.

[0122] In an embodiment of the present invention, the initial expected position coordinate information is obtained by prediction based on the pre-trained position prediction model in the process of determining the shooting prop state of the shooting prop module at the current time point at the previous moment of the current time point; wherein the prediction time range associated with the current time point includes multiple prediction moments, and the initial expected position coordinate information is composed of the output trajectory coordinate points, orientation data and speed data of the shooting prop module at each prediction moment.

[0123] In an embodiment of the present invention, for example, a server uses a pre-trained position prediction model to predict the state of a shooting prop module at the moment before the current time point, and obtains initial expected position coordinate information based on the prediction results. The initial expected position coordinate information includes trajectory coordinate points, orientation data, and velocity data at multiple prediction moments. Consider an online multiplayer shooting game where the server is processing the state of a machine gun module used by an AI character named "AI Character A." The performance of the machine gun module in the game is affected by multiple factors, including player input, the game physics engine, and environmental interaction. Assuming the current time point is the 10th second, at the previous moment, i.e., the end of the 9th second, the server performed a series of calculations and predictions. These calculations are typically performed to maintain real-time responsiveness and smooth animation effects in the game. The server has a pre-trained position prediction model, which may be based on a machine learning algorithm (such as a neural network) and has been trained to predict the position changes of the shooting prop module within a given time range. For example, the model may have learned the behavior patterns of the machine gun module under different AI operations (such as movement, aiming, shooting) and environmental conditions (such as wind speed, terrain, etc.). At the end of the 9th second, the server uses this prediction model to predict the state of the machine gun module's shooting prop at the 10th second (the current time). The predicted state may include information such as the machine gun's expected position, heading, and firing rate. The prediction model outputs a series of data that constitutes the initial expected position coordinate information. This information includes the machine gun module's expected position coordinates at each predicted moment within the 10th second (for example, every 0.1 second), heading data (such as the angle of the muzzle), and velocity data (such as movement speed and firing rate). For example, the prediction model may output the following information: At 10.0 seconds, the machine gun module's position coordinates are (X1, Y1, Z1), heading due north, and firing rate is 600 rounds per minute. At 10.1 seconds, the predicted position is (X2, Y2, Z2), heading slightly toward the northeast, and firing rate remains unchanged. This continues until the end of the predicted time range. This data provides the server with a baseline for calculating the behavior of AI character A's machine gun module in the short term, and provides necessary information for other AI and game systems (such as collision detection, visual rendering, etc.). At the same time, the server will also update and adjust these predictions in real time based on actual game conditions (such as new input from the AI, changes in the game environment, etc.).

[0124] In an embodiment of the present invention, the operation command includes a direction vector and a speed value instructing the shooting prop module to move; the embodiment of the present invention also provides the following implementation methods.

[0125] Based on the direction vector in the operation command, performing a correction operation on the orientation data of the shooting prop module at each prediction moment in the initial expected position coordinate information to obtain a correction direction vector of the shooting prop module at each prediction moment;

[0126] Based on the direction vector and speed value in the operation command, a correction operation is performed on the speed data of the shooting prop module at each predicted moment in the initial expected position coordinate information to obtain a corrected speed value of the shooting prop module at each predicted moment;

[0127] Based on the corrected movement path points and corrected speed values ​​matched at the preceding prediction moments of the plurality of prediction moments, a correction operation is performed on the trajectory coordinate points matched at the target prediction moment of the plurality of prediction moments to obtain the corrected movement path points matched by the shooting prop module at the target prediction moment; the preceding prediction moment is the moment before the target prediction moment;

[0128] The expected position coordinate information is obtained based on the corrected direction vector, corrected speed value and corrected moving path point of the shooting prop module at each predicted moment.

[0129] In an embodiment of the present invention, consider, for example, an online first-person shooter game in which an AI controls a character and wields a shooting gadget module, such as a rifle. The server is responsible for handling the game logic, including AI input, gadget module status updates, and game world physics simulation. The server receives command commands from the AI, which contain direction vectors and velocity values ​​instructing the shooting gadget module to move. For example, the AI ​​might simulate pressing the W, A, S, and D keys on the keyboard to control the character's forward, backward, and left and right movements, while simultaneously simulating mouse movements to control the shooting gadget module's aiming direction. The server first corrects the shooting gadget module's orientation data at each predicted moment in the initial predicted position coordinate information based on the direction vector in the command commands. This means that the server adjusts the gadget module's orientation at each predicted moment to better align with the AI's actual input. If the AI ​​simulates a mouse operation to aim the gadget module in a certain direction, the server adjusts the gadget module's orientation data at each predicted moment based on this input to ensure that the gadget module's aiming direction aligns with the AI's input. Next, the server corrects the velocity data of the shooting prop module at each prediction moment in the initial predicted position coordinate information based on the direction vector and velocity value in the operation command. This involves adjusting the prop module's movement speed and shooting speed at each prediction moment to reflect the AI ​​input. If the AI ​​simulates pressing the W key to move the character forward, the server will increase the prop module's forward speed at each prediction moment based on this input. Similarly, if the AI ​​simulates pressing the S key to move the character backward, the server will decrease the forward speed or increase the backward speed. The server then corrects the trajectory coordinate points matching the target prediction moment based on the corrected movement path points and corrected velocity values ​​from the previous prediction moment (i.e., the moment before the target prediction moment). This means that the server adjusts the prop module's position at the current moment based on the correction results from the previous moment. If the prop module moved a certain distance to the left due to AI input at the previous prediction moment, the server will adjust the prop module's position coordinate points at the current prediction moment based on this movement distance. Finally, the server obtains the corrected expected position coordinate information based on the shooting prop module's corrected direction vector, corrected velocity value, and corrected movement path points at each prediction moment. This information more accurately reflects the prop module's expected future position and behavior after taking into account AI input. After the correction steps above, the server now has an updated and more accurate set of expected position coordinate information, which includes the prop module's expected position, direction, and velocity at each prediction moment. This information is used in subsequent game logic processing, such as collision detection and visual rendering.

[0130] In an embodiment of the present invention, the shooting prop module is configured with multiple dynamic nodes of a virtual structure; the aforementioned step of obtaining the current prop interaction information of the shooting prop module at the current time point can be implemented through the following example.

[0131] Obtain the number of rotation axes matched by each dynamic node of the shooting prop module;

[0132] Calculating the rotation amount of each dynamic node of the shooting prop module at the current time point under each matching rotation axis number;

[0133] The current prop interaction information is generated based on the rotation amount of each dynamic node under each matched rotation axis number.

[0134] In an embodiment of the present invention, consider, for example, an online first-person shooter game in which an AI possesses a complex prop module capable of shooting and having multiple movable parts (e.g., a barrel, a gun body, and a scope), such as an assault rifle with a foldable stock, adjustable scope magnification, and repositionable grip. These movable parts are controlled in the game by multiple virtual dynamic nodes. The server must process the movement of these dynamic nodes in real time to respond to the AI's actions and generate corresponding prop interaction information. First, the shooting prop module is configured with multiple virtual dynamic nodes during the game design phase. These nodes correspond to the different movable parts in the prop module, and each node has its own rotation axis number, which defines the directions in which the node can rotate freely. The scope may have a dynamic node with a rotation axis number that allows it to rotate horizontally and vertically so that the AI ​​can adjust the scope's field of view; the stock may have another dynamic node with a rotation axis number that allows it to fold or unfold in a single direction. When the server needs to process the shooting prop module, it first obtains the rotation axis number matching each dynamic node in the shooting prop module. This information is typically preset during game design and stored in the game's configuration files. The server queries the configuration files and learns that the scope's dynamic node has two rotation axes (one for horizontal and one for vertical rotation), while the stock's dynamic node has one rotation axis (for folding / unfolding). Next, the server calculates the rotation of each dynamic node in the shooting prop module for each matching rotation axis at the current time. This calculation is typically based on AI input and the results of the game's physics engine simulation. If the AI ​​presses a button to adjust the scope's magnification during gameplay, the server calculates the rotation of the scope's dynamic node for each matching rotation axis based on this input. Similarly, if the AI ​​folds the stock, the server calculates the rotation of the stock's dynamic node for each matching rotation axis. Finally, the server generates the current prop interaction information based on the rotation of each dynamic node for each matching rotation axis. This information is a dataset containing the current position and rotation state of all dynamic nodes in the shooting prop module. The server aggregates the rotations of dynamic nodes, such as the scope and stock, into a data structure to form the current prop interaction information. This information is then used to update the state of the prop module in the game screen, handle interactions with other game objects (such as collision detection), and pass it to other game systems that require this data (such as network synchronization, animation rendering, etc.).

[0135] In an embodiment of the present invention, the shooting prop module is configured with multiple dynamic nodes of virtual structure; each dynamic node is configured with a matching number of rotation axes; the aforementioned step of generating the prop operation information of the shooting prop module at the current time point based on the continuous position coordinate information and the current prop interaction information can be implemented through the following example.

[0136] Based on the pre-trained position prediction model, the reference rotation amount of each dynamic node of the shooting prop module is calculated according to the continuous position coordinate information and the current prop interaction information, when the state of each dynamic node of the shooting prop module is transferred from the current time point to a subsequent time point under each matched rotation axis number;

[0137] The prop operation information is generated based on the calculated reference rotation amount of each dynamic node corresponding to each rotation axis number.

[0138] In an embodiment of the present invention, consider, for example, an online shooting game in which an AI-controlled character possesses a shooting gadget module, such as an assault rifle with adjustable components. This rifle is designed in the game with multiple dynamic nodes, each capable of rotating around a specific rotation axis. The server is required to process the movement of these nodes and generate item manipulation information based on continuous position coordinate information and current item interaction information. During the game design process, the shooting gadget module is configured with multiple virtual dynamic nodes. Each dynamic node has a corresponding rotation axis number, which defines the direction and degrees of freedom in which the node can rotate. The rifle's scope can rotate along the vertical axis to adjust the aiming angle, while the grip may rotate along the horizontal axis to change the grip. Each such rotation action corresponds to a dynamic node and one or more rotation axes. When the server processes the shooting gadget module, it uses a pre-trained position prediction model to calculate the reference rotation of each dynamic node for each matching rotation axis based on the continuous position coordinate information and current item interaction information. This calculation takes into account the gadget module's historical motion trajectory, the AI's current input, and the simulation results of the game's physics engine. If the AI ​​adjusts the scope via input devices (such as a mouse or keyboard) at consecutive points in time, the server collects this input data and, combined with a position prediction model, calculates the appropriate rotations for the scope's dynamic nodes along each vertical axis to achieve the AI's intended effect. The server generates item manipulation information based on the calculated reference rotations for each dynamic node along each corresponding rotation axis. This information is a set of instructions that tells the game engine how to update the state of the shooting prop module to implement the AI's intended operation. If it is calculated that the scope needs to be rotated 10 degrees along the vertical axis and the grip needs to be rotated 5 degrees along the horizontal axis, the server will generate item manipulation information containing these rotation instructions. The game engine then uses this information to update the shooting prop module's representation in the game world to ensure the AI's operations are correctly responded to. Through these steps, the server accurately processes the movement of multiple dynamic nodes in the shooting prop module and generates corresponding item manipulation information based on the AI's real-time input and the results of the game physics simulation. This processing ensures that the shooting prop module in the game is highly interactive and realistic, providing players with a more immersive gaming experience.

[0139] In an embodiment of the present invention, the aforementioned step of calculating the reference rotation amount of each dynamic node of the shooting prop module from the current time point state to the subsequent time point of the current time point under each matching number of rotation axes based on the pre-trained position prediction model according to the continuous position coordinate information and the current prop interaction information can be implemented through the following example.

[0140] Based on the pre-trained position prediction model, the continuous position coordinate information and the current prop interaction information, a normal distribution center value is calculated to indicate the dynamic change of each dynamic node from the current time point to a subsequent time point of the current time point under each rotation axis number; the normal distribution center value includes multiple component values, and one component value of the normal distribution center value is used to indicate the dynamic change of a dynamic node on the corresponding rotation axis number;

[0141] Obtaining a target normal distribution determined by the center value of the normal distribution, and randomly sampling the target normal distribution for each aspect of the center value of the normal distribution to obtain a random rotation amount that matches each aspect;

[0142] The component values ​​and random rotation amounts of each aspect are accumulated respectively to obtain the reference rotation amount of each dynamic node under each matched rotation axis number; a reference rotation amount is the accumulated value of the component values ​​and random rotation amounts of the corresponding aspect.

[0143] In an embodiment of the present invention, for example, consider an online shooting game where the server is processing an AI-controlled shooting prop module, such as a rifle with multiple adjustable components (e.g., a scope, a stock, etc.). These components implement various actions in the game through dynamic nodes and rotation axes. The server needs to predict its future state changes based on the AI's operations and the prop module's historical state. The server first uses a pre-trained position prediction model, combined with continuous position coordinate information and current prop interaction information, to calculate the center value of the normal distribution of the dynamic changes of each dynamic node from the current time point to the subsequent time point for each rotation axis. This center value is actually a multidimensional vector, where each component corresponds to the expected change of a specific dynamic node for a specific rotation axis. If the rifle's scope can be adjusted along two rotation axes (for example, horizontally and vertically), the server will calculate a normal distribution center value component for each of these two directions. These components are calculated based on factors such as the AI's historical operations, the scope's current position, and the simulation results of the game's physics engine. Next, the server determines a target normal distribution based on the calculated center value of the normal distribution and randomly draws a rotation value from this distribution for each component (i.e., the change in each dynamic node for each rotation axis). This random drawing process increases the diversity of prediction results and the uncertainty of the game, allowing each AI operation to produce slightly different results. The server may draw a smaller random rotation value for the horizontal rotation of the scope and a larger random rotation value for the vertical rotation. These random values ​​reflect the uncertainty of AI operations and the errors in the game world physics simulation. Finally, the server adds each component value (i.e., each component of the center value of the normal distribution) and its corresponding random rotation value to obtain the final reference rotation value for each dynamic node for each matching rotation axis. This reference rotation value is used to update the state of the shooting prop module and generate corresponding prop operation information. If the server calculates the center value component of the normal distribution for the horizontal direction of the scope to be 5 degrees and the randomly drawn rotation value is -1 degree, the final reference rotation value will be 4 degrees (5 degrees - 1 degree). Similarly, if the vertical center value component is 3 degrees and the randomly selected rotation is 2 degrees, the final reference rotation will be 5 degrees (3 degrees + 2 degrees). The server will use these reference rotations to update the position and status of the scope in the game. Through the above steps, the server can predict its future state changes based on the AI's operations and the historical status of the prop module and generate corresponding prop operation information. This method not only takes into account the AI's intentions and the simulation results of the game's physics engine, but also increases the diversity and fun of the game by introducing randomness.

[0144] In order to more clearly describe the solution provided by the embodiment of the present invention, an overall framework is provided in the following technical solution.

[0145] 1) Entities Encoder:

[0146] Input: This includes information about the current agent and enemy agents, such as health, alignment, range, direction, ray (whether there are obstacles), Cartesian coordinates (xy), and enemy FSM states (shooting, moving, stationary, etc.). These values ​​are normalized to account for instability caused by scale differences.

[0147] Structure: The Entities Encoder uses a multilayer perceptron (MLP) network. The current agent's scalars are encoded in their own MLP, while the scalars of other agents share the same MLP parameters. The feature vectors encoded from all scalar features maintain the same dimension, facilitating subsequent feature fusion. The Entities Encoder architecture encompasses, but is not limited to, other network forms (such as CNN, RNN, and LSTM).

[0148] 2) Target Enemy Selection Module (Attention+Gumbel Softmax Sampling):

[0149] Input: Feature vector encoded by the current agent, feature vector encoded by the enemy agents entity.

[0150] Model Structure: The enemy selection module uses an attention mechanism. The principle is to use the agent's feature encoding as a query and perform a dot product with the encoding of other enemies.

[0151] Assume that the Agent’s feature vector is q and the feature vector of the i-th enemy is k i , assuming there are at most 10 enemies, then i∈{0,1,2,...,9}. Then q and k i The dot product can be expressed as vector multiplication, that is,

[0152]

[0153] The result of matrix multiplication a is an i-dimensional vector, which represents the correlation with each enemy entity. After Softmax normalization, the discrete action probability of selecting each enemy can be obtained, that is,

[0154] π=Softmax(a)=[π1,π2,π3,...,πi ]

[0155] The probability distribution π is obtained. During model inference, random sampling is performed based on the distribution π to determine the enemy selected by the current agent. However, argmax and random sample cannot return gradients. To ensure their differentiability during training, this paper applies Gumbel Softmax Sampling to the training of this module, allowing the neural network to sample the discrete distribution while still being able to calculate gradients for backpropagation. The formula is:

[0156]

[0157] Where k∈{0,1,2,...,i} represents the kth enemy entity among the i enemy entities;

[0158] The parameter τ>0 is called the temperature coefficient, g k is an independent and identically distributed standard Gumbel distributed random variable. g k Generated from a uniform distribution by inverting the Gumbel distribution, i.e.:

[0159] g i =-log(-log(x i )),wh ere x~U(0,1)

[0160] 3) Map Encoder:

[0161] Input: This invention models the in-game map by gridding it (e.g., a 1m x 1m grid as the smallest unit). The input to the network is not the game screen, but rather map information for a square with sides of N meters, centered on the Agent. This map information includes: 1. Normalized terrain height map; 2. Shooting mask; 3. Movement mask; 4. Enemy position information.

[0162] The enemy's position information is mapped from the enemy's coordinates (x, y) to the coordinates (μ x and μ y ), with σ as the center and a 2D Gaussian distribution with standard deviation.

[0163] The Map Encoder is a convolutional neural network (CNN) used to extract features from map feature maps. The Map Encoder of the present invention uses a fully convolutional network (FCN) architecture to preserve spatial location features. The encoder architecture should include, but is not limited to, other networks that process two-dimensional feature maps (e.g., Vgg, ResNet, DenseNet, EfficientNet, Vit, etc.).

[0164] 4) Policy Network (Actor):

[0165] The Actor of the present invention includes decision modules for two actions: shooting and moving. It supports the combination of different action types, that is, it can shoot while moving, and can also be expanded to other action behaviors.

[0166] Mobility Policy Subnetwork:

[0167] The input is a spatial feature map encoded by the Map Encoder, and the output is the agent's movement direction. The spatial feature map input passes through the movement strategy subnetwork, resulting in a predicted 3*3 feature map. Each spatial position in this feature map corresponds to nine possible movement directions for the agent in the actual environment. These nine directions may include forward, backward, left, right, upper left, upper right, lower left, lower right, and staying still. The feature value at each position may represent the probability or confidence that the agent will choose to move in that direction. Therefore, by examining the feature values ​​at each position in this 3*3 feature map, the agent can determine its next movement direction. For example, if the feature value in a certain direction is significantly higher than in other directions, the agent may choose to move in that direction. In this way, the movement strategy subnetwork can guide the agent's movement decisions based on the spatial feature map.

[0168] The feature map is normalized by Softmax and converted into a probability distribution of movement directions. The structure of the movement strategy subnetwork should include but not be limited to other networks that process two-dimensional feature maps (such as Vgg, ResNet, DenseNet, EfficientNet, Vit, etc.).

[0169] Shooting strategy subnetwork:

[0170] The input of this network needs to include the spatial features encoded by Map Encoder, the player state features output by MLP, and the target enemy features obtained by Gumbel sampling.

[0171] The first step is to perform feature fusion on the three input features. This fusion method is to perform element-wise addition on the three features, that is, to add them element by element to obtain the fused feature vector.

[0172] The action space of the shooting strategy sub-network is 2, i.e., the two actions of shooting and not shooting. The fused dimensional feature vector passes through the shooting strategy sub-network to obtain a 2D feature vector, which is then normalized by Softmax to obtain the probabilities of the two action spaces (shooting and not shooting).

[0173] 5) Value Function Network (Critic):

[0174] Critic is used to learn the relationship between the environment and rewards, with the goal of evaluating the actions selected by the Actor and assisting in training.

[0175] The input to the Critic is the output vector of the entity scalar encoder. The network structure encodes the features of k enemy entities, averages the resulting feature vector, and then performs element-wise addition on the feature vector of the current agent to obtain a fused feature vector. This fused feature vector passes through the MLP network, and the final output is the Critic's value, which is the value V(s) of state s, that is, the Q-value of all actions in state s, and the expected value under a specific strategy.

[0176] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned intelligent agent control method based on deep reinforcement learning. Figure 2 As shown, Figure 2 This is a block diagram of the structure of a computer device 100 provided in an embodiment of the present invention. Computer device 100 includes a memory 111, a processor 112, and a communication unit 113. To enable data transmission or exchange, memory 111, processor 112, and communication unit 113 are electrically connected to each other, directly or indirectly. For example, these components can be electrically connected via one or more communication buses or signal lines.

[0177] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the present disclosure and to utilize various embodiments configured with various modifications as appropriate for the specific application contemplated.

Claims

1. An intelligent agent control method based on deep reinforcement learning, characterized in that: include: Acquire current state information of a current agent and hostile state information of multiple hostile agents, and input the current state information of the current agent and the hostile state information of the multiple hostile agents into an entity scalar encoder constructed based on a multi-layer perception mechanism to obtain an entity feature vector of the current agent and hostile feature vectors corresponding to each of the multiple hostile agents; Inputting the entity feature vector and the plurality of hostile feature vectors into a target enemy selection unit constructed based on an attention mechanism, and determining a target hostile agent from the plurality of hostile agents; Obtaining the game map where the current agent is located, and extracting the map coding features of the current agent; Inputting the entity feature vector, the target hostile feature vector corresponding to the target hostile agent, and the map encoding feature into a strategy network to obtain an action command for indicating whether the current agent performs a shooting operation on the target hostile agent; Obtaining a prediction evaluation result of the action command based on a preset value function network; Obtaining an actual reward result of the action command based on a preset reward function; Optimizing and adjusting the policy network according to the predicted evaluation result and the actual reward result, so as to make an action decision of the current agent using the optimized and adjusted policy network; The target enemy selection unit includes a cascaded Softmax architecture and a Gumbel Softmax Sampling architecture, and the entity feature vector and the plurality of hostile feature vectors are input into a target enemy selection unit constructed based on an attention mechanism, and a target hostile agent is determined from the plurality of hostile agents, including: Performing dot product calculation on the entity feature vector and the plurality of hostile feature vectors to obtain a target matrix constructed by multiplying the plurality of vectors, wherein each element in the target matrix is ​​used to characterize the correlation between the current agent and each of the hostile agents; The target matrix is ​​normalized by using the Softmax architecture to obtain a probability distribution corresponding to the target matrix, wherein each element in the probability distribution is used to represent the probability of the current agent selecting a discrete action of each of the hostile agents; Processing the probability distribution using the Gumbel Softmax Sampling architecture to obtain a new probability distribution; In the training phase, the loss function is calculated according to the new probability distribution and back propagation is performed to update the model parameters; In the inference phase, sampling is performed according to the new probability distribution to determine the final selected target hostile agent; The strategy network includes a movement strategy subnetwork and a shooting strategy subnetwork. The entity feature vector, the target hostile feature vector corresponding to the target hostile agent, and the map encoding feature are input into the strategy network to obtain an action command for indicating whether the current agent performs a shooting operation on the target hostile agent, including: Acquire a spatial feature map of the map coding feature, wherein the spatial feature map includes obstacle information of the current agent and position information of the target hostile agent; Inputting the spatial feature map into the mobile strategy subnetwork to obtain a feature map prediction result of a preset size; Normalizing the feature map prediction result to obtain the moving direction probability distribution of the current agent; Performing element-wise addition feature fusion on the entity feature vector, the target hostile feature vector and the map coding feature to obtain a first fused feature vector; Inputting the first fused feature vector into the shooting strategy subnetwork to obtain the action probability distribution of whether the current agent shoots; According to the movement direction probability distribution and the action probability distribution, obtaining an action command for indicating whether the current agent performs a shooting operation on the target hostile agent; The step of obtaining the prediction evaluation result of the action command based on a preset value function network includes: Obtain pending state information of the current agent and pending state information of the plurality of hostile agents after the current agent executes the action command; Extracting the entity state vector of the pending state information of the current agent and the hostile state vectors corresponding to the pending state information of the plurality of hostile agents; Averaging a plurality of the hostile state vectors to obtain an average hostile state vector; Performing element-wise addition feature fusion on the entity state vector and the average enemy state vector to obtain a second fused feature vector; Inputting the second fused feature vector into a preset value function network to obtain a prediction evaluation result of the action command; The obtaining the actual reward result of the action command based on the preset reward function includes: Based on a pre-set reward function: Obtaining the actual reward result of the action command; Among them, the is the actual reward result, Is the reward type The weight of The policy network is obtained by: Acquire an initial policy network, and interact with a preset shooting game environment multiple times based on the initial policy network to obtain a game state trajectory; According to the game state trajectory combined with a preset value function network, the discount rewards and advantage values ​​of the multiple states included in the game state trajectory are calculated; updating the initial policy network based on the discounted reward and the advantage value to obtain an intermediate policy network, and updating the preset value function network based on a mean square loss function; Perform iterative training according to the intermediate strategy network and the updated preset value function network to obtain the trained strategy network; The method of calculating the discount reward and advantage value of the game state trajectory according to the game state trajectory in combination with a preset value function network includes: According to the formula: Calculate the discounted reward of the game state trajectory; wherein, For the said discount reward, The state estimated by the preset value function network The value function is the discount factor, is the number of interactions between the strategy network and the game environment, The current moment of the game; According to the formula: The advantage value of the game state trajectory is calculated; wherein, is the advantage value; The method further comprises: In case the current agent executes an action command of a shooting operation against the target hostile agent, obtaining a shooting prop module configured by the current agent for executing the shooting operation; Obtain continuous position coordinate information of the shooting prop module at the current time point; the continuous position coordinate information represents the position coordinate information of the shooting prop module in the recorded time range and the predicted time range associated with the shooting prop module at the current time point; the continuous position coordinate information is composed of past position coordinate information and expected position coordinate information; wherein the past position coordinate information is the position coordinate information of the recorded time range associated with the shooting prop module at the current time point; the expected position coordinate information is obtained after analyzing the initial expected position coordinate information and the operation command for the shooting prop module at the current time point, and the initial expected position coordinate information is based on the output of the pre-trained position prediction model. The position coordinate information of the shooting prop module in the predicted time range associated with the current time point; the past position coordinate information is generated by: obtaining multiple historical moments in the recorded time range associated with the current time point; obtaining the trajectory coordinate points, direction data and speed data of the shooting prop module at each of the multiple historical moments; generating the past position coordinate information based on the trajectory coordinate points, direction data and speed data of the shooting prop module at each historical moment; the initial expected position coordinate information is obtained by prediction based on the pre-trained position prediction model in the process of determining the shooting prop state of the shooting prop module at the current time point at the previous moment of the current time point; Wherein, the predicted time range associated with the current time point includes multiple predicted moments, the initial expected position coordinate information is composed of the output trajectory coordinate points, orientation data and speed data of the shooting prop module at each predicted moment; the operation command includes a direction vector and a speed value instructing the shooting prop module to move; the expected position coordinate information is obtained in the following manner: based on the direction vector in the operation command, a correction operation is performed on the orientation data of the shooting prop module at each predicted moment in the initial expected position coordinate information to obtain the corrected direction vector of the shooting prop module at each predicted moment; based on the direction vector and the speed value in the operation command, the direction vector in the initial expected position coordinate information is corrected. The shooting prop module performs a correction operation on the rate data at each prediction moment to obtain the corrected speed value of the shooting prop module at each prediction moment; based on the corrected moving path points and corrected speed values ​​matched at the preceding prediction moments of the multiple prediction moments, a correction operation is performed on the trajectory coordinate points matched at the target prediction moment of the multiple prediction moments to obtain the corrected moving path points matched by the shooting prop module at the target prediction moment; the preceding prediction moment is the previous moment of the target prediction moment; based on the corrected direction vector, corrected speed value and corrected moving path points of the shooting prop module at each prediction moment, the expected position coordinate information is obtained; the shooting prop module is configured with multiple dynamic nodes of virtual structure; Obtaining the number of rotation axes matched by each dynamic node of the shooting prop module; Calculate the rotation amount of each dynamic node of the shooting prop module at the current time point under each matching rotation axis number; Based on the rotation amount of each dynamic node under each matching rotation axis number, current prop interaction information is generated; the current prop interaction information represents the shooting prop state of the shooting prop module at the current time point; the shooting prop module is configured with multiple dynamic nodes of virtual structure; each dynamic node is configured with a matching rotation axis number; Based on the pre-trained position prediction model, according to the continuous position coordinate information and the current prop interaction information, the reference rotation amount of each dynamic node of the shooting prop module transferred from the current time point state to the subsequent time point of the current time point under each matched rotation axis number; Based on the calculated reference rotation amount of each dynamic node corresponding to each rotation axis number, generating prop operation information; the prop operation information represents the operation mode of the shooting prop module from the current time point to a subsequent time point of the current time point; Determine the shooting prop state of the shooting prop module at a time point subsequent to the current time point based on the prop operation information and the current prop interaction information, and switch the shooting prop module from the shooting prop state indicated by the current prop interaction information to the shooting prop state at a time point subsequent to the current time point; The position prediction model based on the pre-training is used to calculate the reference rotation amount of each dynamic node of the shooting prop module transferred from the current time point state to the subsequent time point of the current time point under each matched rotation axis number according to the continuous position coordinate information and the current prop interaction information, including: Based on the pre-trained position prediction model, based on the continuous position coordinate information and the current prop interaction information, a normal distribution center value is calculated to indicate the dynamic change of each dynamic node from the current time point to a subsequent time point of the current time point under each rotation axis number; the normal distribution center value includes component values ​​of multiple aspects, and a component value of one aspect of the normal distribution center value is used to indicate the dynamic change of a dynamic node on a corresponding rotation axis number; Obtaining a target normal distribution determined by the center value of the normal distribution, and randomly sampling the target normal distribution for each aspect of the center value of the normal distribution, to obtain a random rotation amount that matches each aspect; The component values ​​and random rotation amounts of each aspect are accumulated respectively to obtain the reference rotation amount of each dynamic node under each matched rotation axis number; a reference rotation amount is the accumulated value of the component values ​​and random rotation amounts of the corresponding aspect.

2. A server system, characterized in that: The method comprises a server, wherein the server is configured to execute the method of claim 1.