Intelligent game role dynamic expansion and cooperative combat method and system for virtual battlefield

Abstract

The invention provides an intelligent game role dynamic extension and cooperative combat method for a virtual battlefield, and relates to the technical field of intelligent games, the method adopts an UpDeT module to dynamically increase or reduce the number or type of agents according to the real-time battlefield condition and strategy requirements of the virtual battlefield; state features and action features corresponding to different agent types of different scenes are converted to a unified feature space through a feature mapping function; based on a self-attention mechanism of Transform, carrying out fusion processing on the feature representation of each agent to obtain a fusion feature; generating a cooperative combat decision according to the fusion feature and the communication information; therefore, the problems of fixed roles, poor expansibility and difficulty in strategy migration in the prior art are solved, the generalization ability of the intelligent game strategy is improved, and the intelligent game level is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent game technology, and in particular to a method and system for dynamic expansion and coordinated combat of intelligent game roles in a virtual battlefield. Background Art

[0002] A virtual battlefield is a computer simulation platform that simulates real battlefield environments and conditions. It is widely used in military training, tactical research, weapon system testing, and strategic planning. These platforms create a virtual environment that simulates real geographical environments, weather conditions, terrain, and battlefield obstacles. Users can interact with the virtual environment by inputting different tactical decisions and commands. The platform will simulate the corresponding battlefield response based on these inputs, allowing users to simulate various military operations without actual risks.

[0003] In addition, virtual battlefields can also be applied to the gaming field. For example, by creating immersive AR and VR environments, players can be provided with a more realistic and intense gaming experience, allowing them to immerse themselves in a simulated battlefield environment and experience the tension and excitement of battle; or, they can be used to design multiplayer online combat games and shooting games, provide dynamic battlefield environments and real-time combat interactions, and enhance competition and cooperation between players; or, they can be used to develop educational games to help players learn tactical knowledge, historical events, and strategic thinking; and so on.

[0004] The virtual battlefield provides an environment, and the intelligent agent is the active participant or agent in this environment. The intelligent agent can be a friendly character, an enemy character, or a neutral character, and they perform tasks and interact in the virtual battlefield. In the virtual battlefield, the intelligent agent needs to make tactical decisions, such as choosing the timing of attack, movement path, and resource allocation. These decisions are usually based on complex algorithms and machine learning models.

[0005] Traditional AI agent development mainly relies on manual design and customized solutions for specific environments. Developers usually write specific agent architectures and codes for a single environment. Common methods include value function approximation based on Q-learning, policy gradient methods, and algorithms such as DQN, PPO, and A3C in deep reinforcement learning. However, this approach has the following limitations in complex and changing application scenarios:

[0006] (1) Lack of a unified design template: The development of each multi-agent service requires redefining the state space, action space, and reward function, which makes the development process cumbersome and difficult to reuse existing designs and codes.

[0007] (2) Poor scalability of game characters: It is impossible to flexibly and dynamically adjust the number of agents or introduce new agent types, which limits the flexibility of AI agents and makes it difficult to adapt to changes in the environment and upgrades in demand;

[0008] (3) Difficulty in agent migration: It is difficult to directly apply an agent that has been trained in a specific environment to other environments. This is because different environmental characteristics, rules, and goals may vary significantly, resulting in insufficient generalization of strategies.

[0009] In addition, some existing reinforcement learning platforms and tools, such as OpenAI Gym and Unity ML-Agents, although providing a basic framework of the environment and algorithms, still require developers to implement specific agent logic on their own, and lack high-level design templates and multi-agent collaboration support.

[0010] Therefore, traditional reinforcement learning agents have the problem of fixed roles when developing high-quality intelligent game systems, and are unable to support seamless changes in the number of roles and coordinated operations between roles after changes. Summary of the invention

[0011] In view of this, the purpose of the present invention is to propose a method and system for dynamic expansion and collaborative combat of intelligent game roles in a virtual battlefield, so as to solve the problems of fixed roles, poor scalability and difficulty in strategy migration in the prior art, improve the generalization ability of intelligent game strategies, and enhance the level of intelligent games.

[0012] Based on the above purpose, on the one hand, the present invention provides a method for dynamic expansion and coordinated combat of intelligent game roles in a virtual battlefield, comprising the following steps:

[0013] According to the real-time battle situation and strategy requirements of the virtual battlefield, the UpDeT module is used to dynamically increase or decrease the number or type of intelligent agents; wherein the intelligent agents represent game roles;

[0014] The state features and action features corresponding to different agent types in different scenes are converted to a unified feature space through a feature mapping function to obtain the feature representation of each agent in the feature space to ensure the consistency of the feature dimension; the feature space includes a state space and an action space;

[0015] Based on the self-attention mechanism of Transformer, the feature representations of each agent are fused and the fused features are obtained by capturing the correlation information between the agents.

[0016] Based on the obtained fusion features, a strategy network is used to generate a collaborative combat decision; wherein the strategy network adopts an extended strategy network, and the strategy network also considers the communication information of the intelligent agent; and a collaborative combat decision is generated according to the fusion features and the communication information.

[0017] As a preferred solution of the present invention, the UpDeT module includes an agent type identification unit, a quantity adjustment strategy unit and an agent instantiation unit, wherein the agent type identification unit is used to identify the agent type required by the current virtual battlefield, the quantity adjustment strategy unit determines the increase or decrease in the number of agents according to the battlefield situation and strategic goals, and the agent instantiation unit is responsible for creating or destroying corresponding agent instances according to the identified type and quantity.

[0018] As a preferred solution of the present invention, the game role intelligent agent includes a perception module, a decision module and a communication module; wherein the perception module is used to acquire and process environmental information and generate spatial features and state features; the decision module uses a strategy network to generate collaborative combat decisions; and the communication module is used to share and interact with other intelligent agents to achieve collaborative decision-making.

[0019] As a preferred solution of the present invention, the fusion processing based on the self-attention mechanism further includes the following steps:

[0020] Extract agent features and set initial weights for each agent feature;

[0021] Using the initialized weights, convert each agent’s features into keys, queries, and values;

[0022] For each agent, use its query representation to perform a dot product operation with the key representation of other agents to obtain an attention score; based on the attention score, determine the degree of association between the current agent and other agents;

[0023] Apply the Softmax function to the attention scores of all agents to normalize the scores into a probability distribution;

[0024] The normalized attention score is used as a weight to perform weighted summation on the value representation of each agent to obtain the fusion feature.

[0025] As a preferred solution of the present invention, the communication mechanism between various agents adopts: defining a unified communication interface for exchanging communication information between various agents; when adding a new agent, registering to the communication interface; when removing an agent, deregistering from the communication interface; the communication information is represented as a vector c:

[0026]

[0027] Where N is the number of agents, mi is the message of the i-th agent, and h is the message encoding function.

[0028] Furthermore, the extended strategy network generates collaborative combat decisions based on the fusion features and communication information, using the following algorithm:

[0029] π=Softmax(fθi(si',ci));

[0030] Among them, π represents the probability of selecting action a in state s, si' represents the fusion feature of agent i, ci represents the communication information of agent i, and θi represents the policy network parameters.

[0031] As a preferred solution of the present invention, the motivation of the agent is also strengthened through a reward scaling mechanism, which specifically includes:

[0032] Set a base reward value for the task;

[0033] Set a unified zoom range according to the game scenario of the virtual battlefield;

[0034] Setting a scaling factor for the task within the scaling range;

[0035] Multiply the base reward value by the scaling factor to get the scaled reward value;

[0036] Use the scaled reward as a feedback signal during the agent training process and update the agent’s policy network or policy function;

[0037] Among them, the rewards only consider positive rewards, encouraging the agent to take behaviors that are conducive to teamwork and goal achievement.

[0038] As a preferred solution of the present invention, it also includes differential processing of dead agents and lost agents generated during the game, which specifically includes the following steps:

[0039] The feature sets of dead agents and lost agents are defined separately: the feature set of dead agents includes death time, death location, death cause, and state before death; the feature set of lost agents includes loss time, location before loss, state before loss, and possible loss cause;

[0040] Encode features of dead agents and missing agents separately using one-hot encoding or embedding representation;

[0041] Adjust the dimensionality by adding zero vectors, truncating vectors, or performing feature selection to ensure that the feature vectors of all agents have the same dimensionality;

[0042] According to the preset order, the feature vectors of each agent are sequentially concatenated into a feature sequence;

[0043] The concatenated feature sequence is used as the input of the policy network.

[0044] As a preferred solution of the present invention, when adding the intelligent agent trained in the original virtual battlefield to the new virtual battlefield, the policy network parameters trained in the source environment are fine-tuned in the target environment by using the policy transfer learning mechanism; the loss function used for the fine-tuning is:

[0045]

[0046] Among them, θ t represents the parameters of the policy network in the target environment, represents the parameters of the policy network that has been trained in the source environment, represents the state representation after feature mapping, and π represents the strategy function, which is the probability distribution function used to generate action a in reinforcement learning; Indicates in feature state Next, by the parameter θ t The probability distribution of the defined strategy for action a; D t Represents the data distribution of the target environment, that is, the state in the target environment The sampling distribution of Refers to the data distribution D from the target environment t Medium sampling state Subsequent error terms are then averaged.

[0047] In addition, multi-task learning and regularization techniques are used to optimize the overall loss function:

[0048]

[0049] Where θ represents the overall parameters of the policy network being trained or fine-tuned, E represents the number of tasks or environments; λ e is the weight coefficient used to control the loss of the e-th task to the total loss L in multi-task learning. total The contribution of L e (θ) represents the loss function of the e-th task or environment; β is a regularization coefficient used to balance the influence between the main loss and the regularization term, where the main loss is the sum of the multi-task losses; is a penalty term, indicating that we hope that the current parameter θ does not deviate too much from the source environment or prior parameters

[0050] Another aspect of the present invention provides a system for dynamic expansion and coordinated combat of intelligent gaming roles in a virtual battlefield, which includes:

[0051] UpDeT module: According to the real-time battle situation and strategy requirements of the virtual battlefield, the UpDeT module is used to dynamically increase or decrease the number or type of intelligent agents; wherein the intelligent agents represent game roles;

[0052] Feature mapping module: converts the state features and action features corresponding to different agent types in different scenes into a unified feature space through a feature mapping function, obtains the feature representation of each agent in the feature space, and ensures the consistency of feature dimensions; the feature space includes a state space and an action space;

[0053] Feature fusion module: Based on the self-attention mechanism of Transformer, the feature representations of each agent are fused and the fused features are obtained by capturing the correlation information between the agents.

[0054] Decision generation module: Based on the obtained fusion features, a strategy network is used to generate a collaborative combat decision; wherein the strategy network adopts an extended strategy network, and the strategy network also considers the communication information of the intelligent agent; and a collaborative combat decision is generated according to the fusion features and the communication information.

[0055] Compared with the prior art, the method based on the invention has the following beneficial effects:

[0056] 1. Through the UpDeT module, the present invention can dynamically increase or decrease the number or type of agents according to the real-time battle situation and strategic requirements of the virtual battlefield. This dynamic management mechanism breaks the limitation of fixed roles in traditional methods and enables the system to flexibly respond to various battlefield changes and strategic adjustments.

[0057] 2. The present invention converts the state features and action features of different agent types into a unified feature space, ensuring the consistency of feature dimensions. This process simplifies the subsequent feature fusion and decision generation steps, and improves the operating efficiency and accuracy of the system. At the same time, the unified feature space also facilitates information sharing and interaction between agents, and promotes the realization of collaborative operations.

[0058] 3. The present invention can capture the correlation information between various agents and generate fusion features through the self-attention mechanism of Transformer. This fusion method not only takes into account the characteristics of the agent itself, but also makes full use of the interaction and influence between agents, thereby improving the decision-making quality and efficiency of collaborative operations.

[0059] 4. The present invention adopts an extended strategy network, which not only considers the fusion characteristics of the intelligent agent, but also introduces communication information as the basis for decision-making. This design enables the intelligent agent to have a more comprehensive understanding of the battlefield situation and the status of teammates, so as to make more reasonable decisions. At the same time, the unified communication interface and flexible registration / deregistration mechanism ensure the timeliness and accuracy of information exchange between intelligent agents, providing strong support for collaborative operations.

[0060] 5. The reward scaling mechanism of the present invention positively incentivizes the behavior of the agent by setting the basic reward value, scaling range and scaling factor. This reward mechanism not only encourages the agent to take behaviors that are conducive to teamwork and goal achievement, but also improves the agent's enthusiasm and participation. At the same time, the flexibility of the reward scaling mechanism enables the system to dynamically adjust according to the battlefield situation and strategic goals, further optimizing the combat effect.

[0061] 6. The present invention performs differentiated processing on dead and lost agents, which not only retains the useful information of dead and lost agents, but also provides comprehensive data support for the input of the policy network. At the same time, the method of splicing feature sequences according to a preset order enables the system to conveniently process multiple agent states, improving the robustness and scalability of the system.

[0062] 7. By using the strategy transfer learning mechanism, the present invention can fine-tune the intelligent agent trained in the original virtual battlefield in the target environment when adding it to the new virtual battlefield. This transfer learning method not only reduces the training time and cost of the intelligent agent in the new environment, but also improves the adaptability and combat efficiency of the intelligent agent to the new environment. At the same time, the introduction of multi-task learning and regularization technology further optimizes the overall loss function and improves the generalization ability and stability of the system.

[0063] 8. The present invention uses the Transformer's self-attention mechanism to fuse the feature representations of the agents, capture the correlation information between the agents, and thus generate collaborative combat decisions. This collaborative combat capability enables the system to more effectively utilize the interactions and influences between agents and improve the overall combat level.

[0064] In summary, the method and system for dynamic expansion and coordinated combat of intelligent game roles in a virtual battlefield proposed by the present invention successfully solves the key problems in the prior art and significantly improves the generalization ability of intelligent game strategies and the level of intelligent game. This innovative achievement is of great significance for promoting the development and application of intelligent game technology in virtual battlefields.

[0065] These and other aspects of the present invention will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described below in conjunction with specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0067] Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Furthermore, the order of steps shown in the embodiments is only an example, and does not necessarily include all contents and operations / steps, nor must it be performed in the order described. For example, some operations / steps may also be decomposed, combined, or partially merged, so the actual execution order may change according to actual conditions.

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, some implementation methods of the present invention are described in detail below in conjunction with specific embodiments. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0070] In order to overcome the problem of fixed roles in traditional reinforcement learning agents when developing high-quality intelligent game systems, and the inability to support the problem of senseless changes in the number of roles and the problem of coordinated operations between roles after the changes, a method for dynamic expansion and coordinated operations of intelligent game roles in a virtual battlefield provided by an embodiment of the present invention includes the following steps:

[0071] According to the real-time battle situation and strategy requirements of the virtual battlefield, the UpDeT module is used to dynamically increase or decrease the number or type of intelligent agents; wherein the intelligent agents represent game roles;

[0072] The state features and action features corresponding to different agent types in different scenes are converted to a unified feature space through a feature mapping function to obtain the feature representation of each agent in the feature space to ensure the consistency of the feature dimension; the feature space includes a state space and an action space;

[0073] Based on the self-attention mechanism of Transformer, the feature representations of each agent are fused and the fused features are obtained by capturing the correlation information between the agents.

[0074] Based on the obtained fusion features, a strategy network is used to generate a collaborative combat decision; wherein the strategy network adopts an extended strategy network, and the strategy network also considers the communication information of the intelligent agent; and a collaborative combat decision is generated according to the fusion features and the communication information.

[0075] The UpDeT (Universal Policy Decoupling Transformer) module is a Transformer-based multi-agent reinforcement learning framework. The core idea of ​​the UpDeT module is to treat a single observation as a set of entities (agents or other non-agent parts), and then divide the action space into multiple subsets corresponding to these entities. In this way, UpDeT can optimize the policy at the action subset level, which is called policy decoupling.

[0076] In this embodiment, the UpDeT module includes:

[0077] The agent type identification unit uses deep learning, machine learning or pattern recognition technology to analyze the tasks, environmental conditions and strategic requirements of the current virtual battlefield and identify the type of agent required by the current virtual battlefield;

[0078] The quantity adjustment strategy unit determines the increase or decrease of the number of agents through real-time analysis of battlefield data, according to the battlefield situation and strategic goals, so as to keep it within a reasonable range, avoiding too many agents leading to waste of resources or too few agents leading to inability to cope with complex battlefield environments;

[0079] The agent instantiation unit creates or destroys corresponding agent instances according to the type and number of identified agents, generates the agent instances using an instantiation engine or template, and configures them to perform specific tasks.

[0080] In this embodiment, the game role agent includes:

[0081] The perception module uses sensor technology, image processing technology, natural language processing technology, etc. to acquire and process environmental information, such as position, speed, direction, obstacles, enemy positions, etc., and generates spatial features and state features;

[0082] The decision-making module uses the strategy network to generate collaborative combat decisions based on the spatial features and state features provided by the perception module and the information of other intelligent agents received by the communication module;

[0083] The communication module is used to share information and interact with other intelligent agents to achieve collaborative decision-making; the communication module may use wireless network communication, wired communication or a specific communication protocol to achieve information exchange between intelligent agents. In addition, it may also use distributed computing technology to coordinate the behavior of multiple intelligent agents, allowing intelligent agents to transmit information, share data and make collaborative decisions to achieve more efficient collaborative operations.

[0084] In this embodiment, the fusion processing based on the self-attention mechanism further includes the following steps:

[0085] Extract agent features and set initial weights for each agent feature; extract useful features from the observation data of each agent. These features may include physical properties such as position, speed, acceleration, direction, as well as advanced information such as the agent's historical behavior and strategic tendencies;

[0086] Using the initialization weights, the features of each agent are converted into three representations: key, query, and value. The initialization weights can be randomly generated or set based on some prior knowledge or strategy. The three representations are the core components of the self-attention mechanism, and the interaction between them will determine the degree of association between agents and the formation of fusion features. Among them, the key is used to match the queries of other agents to calculate the attention score; the query is used to actively find other agents related to itself; the value represents the specific information or features of the agent and will be used for the final weighted summation.

[0087] For each agent, use its query representation to perform a dot product operation with the key representation of other agents to obtain an attention score; the degree of association between the current agent and other agents is determined based on the attention score; the higher the score, the stronger the correlation between the two agents;

[0088] Apply the Softmax function to the attention scores of all agents to normalize the scores into probability distributions. The function converts the scores into probability distributions.

[0089] Using the normalized attention score as the weight, the value representation of each agent is weighted and summed to obtain the fusion feature; the fusion feature contains the key information of all agents and reflects the degree of association and the potential for collaborative operations between them. This feature can be used in subsequent tasks such as strategy formulation and decision optimization.

[0090] In this embodiment, the communication mechanism between the agents adopts: defining a unified communication interface for exchanging communication information between the agents; when adding a new agent, registering to the communication interface; when removing an agent, deregistering from the communication interface; the communication information is represented as a vector c:

[0091]

[0092] Where N is the number of agents, mi is the message of the i-th agent, and h is the message encoding function.

[0093] The unified communication interface has standardized methods and data structures to ensure that different agents can correctly understand and process the information sent to each other. For example, it is necessary to define basic functions such as sending information, receiving information, registering new agents, and deregistering agents. Agents can send information to other agents through the communication interface. The information sent should conform to the data structure defined by the interface and contain necessary information content, sender identifier, receiver identifier, etc. Agents can also receive information from other agents through the communication interface. The received information will be parsed and passed to the corresponding processing function or method for processing. Due to the use of a unified communication interface, the system can easily add or remove agents without major modifications to the existing communication logic. This helps to improve the flexibility and maintainability of the system.

[0094] In this embodiment, the extended strategy network generates a collaborative combat decision based on the fusion features and communication information, using the following algorithm:

[0095] π=Softmax(fθi(si',ci));

[0096] Among them, π represents the probability of selecting action a in state s, si' represents the fusion feature of agent i, ci represents the communication information of agent i, θi represents the policy network parameters, and fθi represents the policy network of the agent.

[0097] The algorithm's workflow is as follows:

[0098] Input processing: First, the fusion feature si' and communication information ci of agent i are used as the input of the policy network fθi;

[0099] Policy network calculation: The policy network fθi is calculated based on the input fusion features and communication information, and generates a real number vector as output;

[0100] Softmax conversion: The output of the policy network is converted through the Softmax function to obtain a probability distribution π;

[0101] Selection of action strategy: According to the probability distribution π, agent i selects an action a to perform; the selection of this action can be achieved through random sampling, greedy strategy or other methods.

[0102] Through this algorithm, the extended policy network is able to utilize fused features and communication information to generate collaborative combat decisions, thereby achieving effective collaboration among various intelligent agents in the virtual battlefield.

[0103] In this embodiment, the agent's enthusiasm is also strengthened through a reward scaling mechanism, which specifically includes:

[0104] Set a base reward value for the task;

[0105] Set a unified zoom range according to the game scenario of the virtual battlefield;

[0106] Setting a scaling factor for the task within the scaling range;

[0107] Multiply the base reward value by the scaling factor to get the scaled reward value;

[0108] Use the scaled reward as a feedback signal during the agent training process and update the agent’s policy network or policy function;

[0109] Among them, the rewards only consider positive rewards, encouraging the agent to take behaviors that are conducive to teamwork and goal achievement.

[0110] By introducing a reward scaling mechanism, we can more accurately evaluate the performance of agents in virtual battlefield game scenarios and motivate them to take actions that are more beneficial to the overall goal and teamwork. This method helps to improve the initiative and collaboration of agents, and provides strong support for intelligent games and collaborative operations in virtual battlefields. At the same time, a mechanism that only considers positive rewards can also help prevent agents from taking overly aggressive or unfavorable actions for teamwork.

[0111] In this embodiment, the dead agents and lost agents generated during the game are also processed differently, which specifically includes the following steps:

[0112] The feature sets of dead agents and lost agents are defined separately: the feature set of dead agents includes death time, death location, death cause, and state before death; the feature set of lost agents includes loss time, location before loss, state before loss, and possible loss cause;

[0113] Encode features of dead agents and missing agents separately using one-hot encoding or embedding representation;

[0114] Adjust the dimensionality by adding zero vectors, truncating vectors, or performing feature selection to ensure that the feature vectors of all agents have the same dimensionality;

[0115] According to the preset order, the feature vectors of each agent are sequentially concatenated into a feature sequence;

[0116] The concatenated feature sequence is used as the input of the policy network.

[0117] In this embodiment, when adding the trained agent in the original virtual battlefield to the new virtual battlefield, the policy network parameters trained in the source environment are fine-tuned in the target environment using the policy transfer learning mechanism, so that the policy of the target environment is as close as possible to the policy distribution learned in the source environment, and the original experience can be quickly used in the new environment. The loss function used for the fine-tuning is:

[0118]

[0119] Among them, θ t Represents the parameters (e.g., weights) of the policy network in the target environment. When performing transfer learning, we need to fine-tune or train the policy parameters in the target environment. Represents the parameters of the policy network that has been trained in the source environment, usually the better policy we learned in a similar or related environment; It represents the state representation after feature mapping. Because the original state spaces of different environments are often inconsistent, it is necessary to transform the states of the source environment and the target environment into a unified or similar feature space through certain mapping or abstraction to facilitate the migration of strategies. π represents the strategy function, which is the probability distribution function used to generate action a in reinforcement learning. Indicates in feature state Next, by the parameter θ t The probability distribution of the defined strategy for action a; D t Represents the data distribution of the target environment, that is, the state in the target environment When we collect data such as state, action, and reward in the target environment, these data follow D t ; Refers to the data distribution D from the target environment t Medium sampling state The subsequent error terms are then averaged, where E represents the mathematical expectation; (·) 2 It represents the mean square error (MeanSquaredError) form, which is used to measure the gap between two distributions or outputs.

[0120] Fine-tuning refers to making small adjustments to some parameters while keeping most parameters unchanged to adapt to the characteristics of the target environment. The loss function is calculated based on the data distribution in the target environment and the output of the policy network, which can reflect the performance of the policy network in the target environment and guide the fine-tuning of parameters.

[0121] The fine-tuning implementation steps include:

[0122] Data collection: Collect data such as the agent's state, action, and reward in the target environment;

[0123] Loss calculation: Calculate the value of the loss function based on the collected data and the output of the current policy network;

[0124] Parameter update: Update the parameters of the policy network based on the gradient information of the loss function;

[0125] Iterative optimization: Repeat the above steps until the performance of the policy network in the target environment reaches stability or meets the preset stopping conditions.

[0126] The policy transfer learning mechanism enables the agent to quickly adapt to the new virtual battlefield environment by fine-tuning the policy network parameters trained in the source environment. This mechanism not only improves the adaptability and generalization ability of the agent, but also reduces the cost and time of retraining.

[0127] In this embodiment, multi-task learning and regularization techniques are also used to optimize the overall loss function; the combination of the two can achieve efficient cross-environment or multi-scenario reinforcement learning, reduce the cost of repeated training, and ensure that the strategy has good robustness and transferability. The overall loss function is as follows:

[0128]

[0129] Where θ represents the overall parameters of the policy network being trained or fine-tuned (similar to θt above, but here it is more inclined to the concept of multi-task learning or multi-environment learning, and θ can adapt to multiple tasks / environments at the same time); E represents the number of tasks or environments, that is, when performing multi-task learning, there are usually multiple different but related tasks or environments that need to be taken into account at the same time; λ e is the weight coefficient used to control the loss of the e-th task to the total loss L in multi-task learning. total The contribution of L e (θ) represents the loss function of the e-th task or environment. For example, in reinforcement learning, L e (θ) may be the policy gradient loss, value function loss or Q-learning loss corresponding to the task; β is the regularization coefficient, which is used to balance the influence between the main loss and the regularization term. The main loss is the sum of the multi-task losses. is a penalty term, indicating that we hope that the current parameter θ does not deviate too much from the source environment or prior parameters By minimizing To a certain extent, it can prevent the model from "forgetting" previously learned strategies, reduce the risk of overfitting, and improve generalization ability.

[0130] Multi-task learning (MTL) is a machine learning method that aims to improve the performance of the model by learning multiple related tasks simultaneously. In complex environments such as virtual battlefields, agents may need to handle multiple tasks at the same time, such as target recognition, path planning, teamwork, etc. Through multi-task learning, agents can use shared representations between different tasks to improve learning efficiency and performance. In multi-task learning, the overall loss function usually contains loss terms for multiple tasks, one loss function for each task. These loss terms are combined into the overall loss function through weighted summation.

[0131] Regularization technology is an effective way to prevent model overfitting. In multi-task learning, regularization terms are used to constrain the complexity of model parameters to avoid the problem that the model performs too well on training data but has poor generalization ability on test data. Common regularization techniques include L1 regularization and L2 regularization.

[0132] L1 regularization: By adding the sum of the absolute values ​​of the model parameters as a regularization term, the model parameters can be encouraged to be sparse, that is, some parameters are zero, thereby improving the generalization ability of the model.

[0133] L2 regularization: By adding the square root of the sum of squares of model parameters as a regularization term, the scale of model parameters can be constrained to prevent overfitting caused by excessively large parameters.

[0134] Corresponding to the method embodiment, the present invention also provides a virtual battlefield intelligent game role dynamic expansion and collaborative combat system, which includes:

[0135] UpDeT module: According to the real-time battle situation and strategy requirements of the virtual battlefield, the UpDeT module is used to dynamically increase or decrease the number or type of intelligent agents; wherein the intelligent agents represent game roles;

[0136] Feature mapping module: converts the state features and action features corresponding to different agent types in different scenes into a unified feature space through a feature mapping function, obtains the feature representation of each agent in the feature space, and ensures the consistency of feature dimensions; the feature space includes a state space and an action space;

[0137] Feature fusion module: Based on the self-attention mechanism of Transformer, the feature representations of each agent are fused and the fused features are obtained by capturing the correlation information between the agents.

[0138] Decision generation module: Based on the obtained fusion features, a strategy network is used to generate a collaborative combat decision; wherein the strategy network adopts an extended strategy network, and the strategy network also considers the communication information of the intelligent agent; and a collaborative combat decision is generated according to the fusion features and the communication information.

[0139] The system includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the method of any one of the above embodiments is implemented.

[0140] The system includes a processor and a memory, and may also include an input system and an output system. The processor, memory, input system and output system may be connected via a bus or other means, and the input system may receive input digital or character information and generate signal input of the method of the present invention. The output system may include display devices such as a display screen.

[0141] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method of the embodiment of the present invention. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of the method of the present invention, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In the present embodiment, the processor is used to run the program code stored in the memory or process data. The processors of multiple computer devices of the computer device of the present embodiment execute various functional applications and data processing of the server by running non-volatile software programs, instructions and modules stored in the memory, i.e., implementing the steps of the above method embodiment.

[0143] It should be understood that, in the absence of mutual conflict, all embodiments, features and advantages described above for the method according to the present invention are also applicable to the method and storage medium of the present invention.

[0144] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0145] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0146] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0147] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention is limited to these examples; under the idea of ​​the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for dynamic expansion and coordinated combat of intelligent game roles in a virtual battlefield, characterized in that: The following steps are involved: According to the real-time battle situation and strategy requirements of the virtual battlefield, the UpDeT module is used to dynamically increase or decrease the number or type of intelligent agents; wherein the intelligent agents represent game roles; The state features and action features corresponding to different agent types in different scenes are converted to a unified feature space through a feature mapping function to obtain the feature representation of each agent in the feature space to ensure the consistency of the feature dimension; the feature space includes a state space and an action space; Based on the self-attention mechanism of Transformer, the feature representations of each agent are fused and the fused features are obtained by capturing the correlation information between the agents. Based on the obtained fusion features, a strategy network is used to generate a collaborative combat decision; wherein the strategy network adopts an extended strategy network, and the strategy network also considers the communication information of the intelligent agent; and a collaborative combat decision is generated according to the fusion features and the communication information.

2. The method according to claim 1, characterized in that The UpDeT module includes an agent type identification unit, a quantity adjustment strategy unit and an agent instantiation unit, wherein the agent type identification unit is used to identify the agent type required in the current virtual battlefield, the quantity adjustment strategy unit determines the increase or decrease in the number of agents according to the battlefield situation and strategic goals, and the agent instantiation unit is responsible for creating or destroying corresponding agent instances according to the identified type and quantity.

3. The method according to claim 1, characterized in that The game role intelligent agent includes a perception module, a decision module and a communication module; wherein the perception module is used to acquire and process environmental information and generate spatial features and state features; the decision module uses a strategy network to generate collaborative combat decisions; and the communication module is used to share information and interact with other intelligent agents to achieve collaborative decision-making.

4. The method according to claim 1, characterized in that: The fusion processing based on the self-attention mechanism further includes the following steps: Extract agent features and set initial weights for each agent feature; Using the initialized weights, convert each agent’s features into keys, queries, and values; For each agent, use its query representation to perform a dot product operation with the key representation of other agents to obtain an attention score; based on the attention score, determine the degree of association between the current agent and other agents; Apply the Softmax function to the attention scores of all agents to normalize the scores into a probability distribution; The normalized attention score is used as a weight to perform weighted summation on the value representation of each agent to obtain the fusion feature.

5. The method according to claim 1, characterized in that The communication mechanism between the agents adopts: defining a unified communication interface for exchanging communication information between the agents; when adding a new agent, registering it to the communication interface; when removing an agent, deregistering it from the communication interface; the communication information is represented as a vector c: Where N is the number of agents, mi is the message of the i-th agent, and h is the message encoding function.

6. The method according to claim 5, characterized in that The extended strategy network generates collaborative combat decisions based on fusion features and communication information, using the following algorithm: π=Softmax(fθi(si',ci)); Among them, π represents the probability of selecting action a in state s, si' represents the fusion feature of agent i, ci represents the communication information of agent i, and θi represents the policy network parameters.

7. The method according to claim 1, characterized in that The agent’s motivation is also strengthened through a reward scaling mechanism, including: Set a base reward value for the task; Set a unified zoom range according to the game scenario of the virtual battlefield; Setting a scaling factor for the task within the scaling range; Multiply the base reward value by the scaling factor to get the scaled reward value; Use the scaled reward as a feedback signal during the agent training process and update the agent’s policy network or policy function; Among them, the rewards only consider positive rewards, encouraging the agent to take behaviors that are conducive to teamwork and goal achievement.

8. The method according to claim 1, characterized in that: It also includes differentiated processing of dead agents and lost agents generated during the game, specifically including the following steps: The feature sets of dead agents and lost agents are defined separately: the feature set of dead agents includes death time, death location, death cause, and state before death; the feature set of lost agents includes loss time, location before loss, state before loss, and possible loss cause; Encode features of dead agents and missing agents separately using one-hot encoding or embedding representation; Adjust the dimensionality by adding zero vectors, truncating vectors, or performing feature selection to ensure that the feature vectors of all agents have the same dimensionality; According to the preset order, the feature vectors of each agent are sequentially concatenated into a feature sequence; The concatenated feature sequence is used as the input of the policy network.

9. The method according to any one of claims 1 to 8, characterized in that: It also includes adding the intelligent agent trained in the original virtual battlefield to the new virtual battlefield, using the strategy transfer learning mechanism to fine-tune the policy network parameters trained in the source environment in the target environment; the loss function used for the fine-tuning is: Among them, θ t represents the parameters of the policy network in the target environment, represents the parameters of the policy network that has been trained in the source environment, represents the state representation after feature mapping, and π represents the strategy function, which is the probability distribution function used to generate action a in reinforcement learning; Indicates in feature state Next, by the parameter θ t The probability distribution of the defined strategy for action a; D t Represents the data distribution of the target environment, that is, the state in the target environment The sampling distribution of Refers to the data distribution D from the target environment t Medium sampling state Subsequent error terms are then averaged; In addition, multi-task learning and regularization techniques are used to optimize the overall loss function: Where θ represents the overall parameters of the policy network being trained or fine-tuned, E represents the number of tasks or environments; λ e is the weight coefficient used to control the loss of the e-th task to the total loss L in multi-task learning. total The contribution of L e (θ) represents the loss function of the e-th task or environment; β is a regularization coefficient used to balance the influence between the main loss and the regularization term, where the main loss is the sum of the multi-task losses; is a penalty term, indicating that we hope that the current parameter θ does not deviate too much from the source environment or prior parameters 10. A virtual battlefield intelligent game role dynamic expansion and collaborative combat system, characterized in that: include: UpDeT module: According to the real-time battle situation and strategy requirements of the virtual battlefield, the UpDeT module is used to dynamically increase or decrease the number or type of intelligent agents; wherein the intelligent agents represent game roles; Feature mapping module: converts the state features and action features corresponding to different agent types in different scenes into a unified feature space through a feature mapping function, obtains the feature representation of each agent in the feature space, and ensures the consistency of feature dimensions; the feature space includes a state space and an action space; Feature fusion module: Based on the self-attention mechanism of Transformer, the feature representations of each agent are fused and the fused features are obtained by capturing the correlation information between the agents. Decision generation module: Based on the obtained fusion features, a strategy network is used to generate a collaborative combat decision; wherein the strategy network adopts an extended strategy network, and the strategy network also considers the communication information of the intelligent agent; and a collaborative combat decision is generated according to the fusion features and the communication information.

Images