A decision-making learning method, device, equipment and medium for commanding intelligent agent

Abstract

The present invention discloses a decision-making learning method, device, equipment and medium for commanding intelligent agents. By constructing commanding intelligent agent groups, each commanding intelligent agent group includes a global decision-making module and a local decision-making intelligent agent, and the local decision-making intelligent agent includes a first type of intelligent agent and a second type of intelligent agent; the commanding intelligent agent group includes a commanding intelligent agent group of a first antagonist and a commanding intelligent agent group of a second antagonist; the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent are respectively configured; the commanding intelligent agent group of the first antagonist and the commanding intelligent agent group of the second antagonist are subjected to confrontation simulation, and the autonomous behavior decision models of the local decision-making intelligent agents respectively included in the commanding intelligent agent group of the first antagonist and the commanding intelligent agent group of the second antagonist are alternately trained, and after the training is completed, an intelligent agent decision model suitable for group confrontation is obtained, thereby improving the decision accuracy of the intelligent agent decision model.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative decision-making modeling, and in particular to a decision-making learning method, device, equipment and medium for commanding an intelligent agent. Background Art

[0002] Plan, optimize and simulate multi-formation joint operations, and present the planned content, battlefield environment, comprehensive battlefield situation, process, etc. in a more concrete way, so as to facilitate the pre-verification of planned plans, task plans, etc., and make timely adjustments and optimizations.

[0003] At present, the current multi-formation joint operations usually form a multi-formation collaborative autonomous behavior decision-making model based on QMIX, and adopt the MaxEnt IRL algorithm, introduce an optional strategy sampler, and iteratively optimize and train the multi-formation collaborative autonomous behavior decision-making model to obtain the optimized reward function and decision-making strategy.

[0004] However, the existing multi-formation joint combat decision-making model only supports the coordinated confrontation simulation of multi-formation joint operations in air strikes. In the face of large-scale decision-making problems, it is difficult to make effective decisions while coordinating multi-formation joint operations, and the decision-making accuracy and generalization performance are poor. Summary of the invention

[0005] The present invention provides a decision learning method, device, equipment and medium for commanding an intelligent agent to solve the problem of poor performance of large-scale global decision making.

[0006] According to one aspect of the present invention, a decision-making learning method for a commanding agent is provided, comprising:

[0007] Constructing command agent groups, each command agent group includes a global decision module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, each type of agent includes at least one agent group; the command agent group includes a command agent group of the first antagonist and a command agent group of the second antagonist;

[0008] respectively configuring an autonomous behavior decision model of a first type of intelligent agent and an autonomous behavior decision model of a second type of intelligent agent;

[0009] A confrontation simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist. During the confrontation simulation, the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist are alternately trained. After the training is completed, an agent decision model suitable for group confrontation is obtained.

[0010] According to another aspect of the present invention, there is provided a decision-making learning device for commanding an intelligent agent, comprising:

[0011] A command agent group construction module is used to construct a command agent group, each command agent group includes a global decision module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of the first antagonist and a command agent group of the second antagonist;

[0012] An autonomous behavior decision model configuration module, used to configure the autonomous behavior decision model of the first type of intelligent body and the autonomous behavior decision model of the second type of intelligent body respectively;

[0013] The confrontation simulation module is used to perform confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation process, alternately train the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist, and after completing the training, obtain an agent decision model suitable for group confrontation.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the decision-making learning method of the command intelligent agent in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to implement the decision-making learning method of the command intelligent agent of any embodiment of the present invention when executed by a processor.

[0019] The technical solution of this embodiment is to construct a command agent group, each command agent group includes a global decision-making module and a local decision-making agent, the local decision-making agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent grouping; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; an autonomous behavior decision model of the first type of agent and an autonomous behavior decision model of the second type of agent are respectively configured; a confrontation simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation, the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist are alternately trained. After the training is completed, an agent decision model suitable for group confrontation is obtained, which can realize collaborative decision-making of multiple types of agents, and solves the problem of poor performance of large-scale global decisions by decomposing global decisions into local decisions, improves the decision accuracy of the agent decision model, and helps to improve the reference value of the decision of the command agent.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a decision-making learning method for a command intelligent agent provided in Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of the structure of a command agent group provided by Embodiment 1 of the present invention;

[0024] Figure 3 is a flowchart of a countermeasure simulation process provided by Embodiment 1 of the present invention;

[0025] Figure 4 is a flow chart of a decision-making learning method for a command intelligent agent provided in Embodiment 2 of the present invention;

[0026] Figure 5 is a flow chart of an autonomous behavior decision model training process provided by Embodiment 2 of the present invention;

[0027] Figure 6 is a structural diagram of a decision-making learning device for commanding an intelligent agent provided in Embodiment 3 of the present invention;

[0028] Figure 7 It is a schematic diagram of the structure of an electronic device for implementing the decision-making learning method of a command intelligent agent according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 It is a flow chart of a decision-making learning method for a command intelligent agent provided in Example 1 of the present invention. This embodiment is applicable to the situation of collaborative decision-making of intelligent agent formations. The method can be executed by a decision-making learning device for the command intelligent agent. The decision-making learning device for the command intelligent agent can be implemented in the form of hardware and / or software. The decision-making learning device for the command intelligent agent can be configured in electronic devices such as computers and servers.

[0033] In this embodiment, the command agent is an intelligent and autonomous entity that can perceive the environment, learn and adapt, make decisions and perform actions. The command agent formation includes one or more command agent groups, each of which includes one or more stand-alone agents, wherein a stand-alone agent refers to an independent command agent; the command agent group can perform complex local tasks through collaboration and coordination between stand-alone agents; the command agent formation performs complex global tasks through collaboration and coordination between command agent groups. Global tasks are tasks that achieve the overall goal of the overall combat strategy, such as destroying the opponent's base and seizing strategic points; local tasks are tasks that achieve the goal of the local combat strategy, such as clearing any stand-alone agent of the opponent. Exemplarily, the command agent group can be an air strike group command agent, and the stand-alone agent in the air strike group command agent is an aircraft; the command agent group can also be a reconnaissance and strike mixed agent group, and the stand-alone agent in the reconnaissance and strike mixed agent group is a surface ship.

[0034] like Figure 1 As shown, the method includes:

[0035] S110. Construct command agent groups, each command agent group includes a global decision-making module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of the first antagonist and a command agent group of the second antagonist.

[0036] In this embodiment, the command agent group is a combination of command agent formations that can conduct coordinated operations. Exemplarily, in the case of a two-party combat, a command agent group is constructed for each party. The global decision-making module is a module for making global decisions, and plays the role of a formation commander in the command agent group. Among them, the global decision is a decision on the execution of global tasks by a single agent, including but not limited to the division of agent formations and the planning of attack tasks for agent formations. The local decision-making agent is a command agent for making local decisions, and plays the role of a formation commander in the command agent group. Among them, the local decision is a decision on the execution of local tasks by a single agent, including but not limited to the planning of attack tasks for single agents in formations.

[0037] Specifically, based on the scenario of collaborative decision-making of intelligent agent formations, the number of opponents is determined, and multiple stand-alone agents are instantiated for each opponent. When the number of opponents is 2, any one opponent is the first opponent, and the other opponent is the second opponent. For each opponent, a command agent group is obtained by constructing a global decision-making module and at least one local decision-making agent, and each type of agent includes one or more agent groups. Optionally, the global decision-making module is created based on expert rules, which is used to compare the target distance of the opponent with the preset distance to determine the type of command grouping task; the type of grouping task includes a first type and a second type; the first type of agent is used to perform the first type of grouping task, and the second type of agent is used to perform the second type of grouping task.

[0038] In this embodiment, the expert rule is one or more rules set by experts in the field of multi-agent collaborative decision modeling technology based on existing knowledge and experience, wherein each rule includes a condition-behavior structure, for example, the structure of the rule can be an "if, then" structure, when the condition of the rule is met, the rule is triggered, and then the behavior is executed. The creation of the global decision module is achieved by deploying the expert rules into the global decision module.

[0039] Specifically, the target distance is the distance between the single-machine intelligent agent of the opponent and the position of the own side. Exemplarily, the target distance can be a spatial distance or a horizontal plane distance, which is not limited in this embodiment. Among them, the own side can be the first opponent, and the opponent is the second opponent; the own side can also be the second opponent, and the opponent is the first opponent. The global decision module detects the target distance in real time and compares the target distance with the preset distance, wherein the preset distance is the target distance pre-set in the expert rules, which is used to determine the type of command and formation task. Among them, the type of command and formation task is the type of global task pre-set in the expert rules. It can be understood that each type of command and formation task corresponds to a type of local decision-making intelligent agent.

[0040] In the case where the types of grouping tasks include the first type and the second type, the local decision-making agent includes the first type agent and the second type agent. The first type agent can execute the first type of grouping tasks, and the second type agent can execute the second type of grouping tasks. Optionally, the first type agent is a long-range strike grouping command agent, and the second type agent is a medium- and short-range strike grouping command agent.

[0041] In this embodiment, the long-range strike formation command intelligent agent is one or more command intelligent agents that can execute long-range strike formation tasks, for example, the long-range strike formation command intelligent agent can be an air strike formation command intelligent agent; the medium-range and short-range strike formation command intelligent agent is one or more command intelligent agents that can execute medium-range and short-range strike formation tasks, for example, the medium-range and short-range strike formation command intelligent agent can be a reconnaissance and strike mixed formation command intelligent agent.

[0042] The technical solution of this embodiment is that the local decision-making intelligent agent includes a long-range strike group command intelligent agent and a medium-range strike group command intelligent agent, which can achieve comprehensive decision-making for single-machine intelligent agents that support long-range strikes and medium-range strikes, and help improve combat efficiency.

[0043] By establishing a communication connection between the global decision-making module and the local decision-making agent of each opponent, the instructions of the global decision-making module are transmitted to the first type of agent and / or the second type of agent in the local decision-making agent. Exemplarily, assuming that the first type of agent is an air strike formation command agent, and the second type of agent is a reconnaissance and strike mixed formation command agent, when the target distance is greater than the preset distance, it can be considered that the opponent's single-machine agent needs to be intercepted at a long distance, and the type of the formation task is determined to be the first type, and the global decision-making module sends a long-range strike instruction to the first type of agent, so that the first type of agent performs the first type of formation task; when the target distance is less than or equal to the preset distance, it can be considered that the opponent's single-machine agent needs to be intercepted at a medium distance, and the type of the formation task is determined to be the second type, and the global decision-making module sends a medium- and short-range strike instruction to the second type of agent, so that the second type of agent performs the second type of formation task.

[0044] For example, Figure 2 Schematic diagram of the structure of a command agent group provided by the first embodiment of the present invention. Figure 2 As shown, the global decision module sends long-range attack instructions to the first type of intelligent agent and sends medium- and short-range attack instructions to the second type of intelligent agent.

[0045] The technical solution of this embodiment creates a global decision-making module based on expert rules and constructs a first type of intelligent agent and a second type of intelligent agent, which enables each type of intelligent agent to perform a corresponding type of grouping task, and breaks down the global task into local tasks corresponding to each type of intelligent agent, which helps to improve the decision-making efficiency of the command intelligent agent group.

[0046] In some embodiments, optionally, the conditions of the rule include at least part of the target distance, the number of targets, and the current firepower resources, etc., wherein the target number is the number of single-machine intelligent agents of the opponent, and the current firepower resources are the firepower resources currently available to the own side. Determining the type of the command formation task based on at least part of the target distance, the number of targets, and the current firepower resources, etc., rather than only based on the target distance, can more accurately determine the type of the command formation task.

[0047] S120, respectively configure an autonomous behavior decision model for the first type of intelligent agent and an autonomous behavior decision model for the second type of intelligent agent.

[0048] In this embodiment, the autonomous behavior decision model is a model for commanding a single agent to make autonomous behavior decisions. By configuring the autonomous behavior decision model corresponding to each type of agent in the agent of that type, each type of agent can autonomously execute the corresponding type of grouping task sent to the agent of that type by the global decision module.

[0049] Exemplarily, assuming that the command agent group of the first antagonist includes 10 first-type agents and 5 second-type agents, the autonomous behavior decision-making model corresponding to the first-type agents in the first antagonist is configured to the 10 first-type agents of the first antagonist, and the autonomous behavior decision-making model corresponding to the second-type agents in the first antagonist is configured to the 5 second-type agents of the first antagonist.

[0050] S130, conduct confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation, alternately train the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist, and after the training is completed, obtain an agent decision model suitable for group confrontation.

[0051] In this embodiment, the confrontation simulation is a simulation of a confrontation between a command agent group of a first antagonist and a command agent group of a second antagonist, and the confrontation simulation goal of each antagonist's command agent group is to defeat the command agent group of the other antagonist.

[0052] Specifically, the confrontation simulation includes multiple training rounds. In two adjacent training rounds, the autonomous behavior decision model of each local decision-making agent in the command agent group of the first antagonist and the autonomous behavior decision model of each local decision-making agent in the command agent group of the second antagonist are alternately trained. The autonomous behavior decision model corresponding to the command agent group of the first antagonist may be trained first, or the autonomous behavior decision model corresponding to the command agent group of the second antagonist may be trained first. This embodiment does not limit this. Optionally, the command agent group of the second antagonist or the local decision-making agent in the command agent group of the first antagonist is determined in an alternating manner as a trainable agent, and the confrontation simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist. During the simulation process, the autonomous behavior decision model of the trainable agent is trained until the training end condition is met and the trainable agent is updated.

[0053] Specifically, determine the opponent to which the command agent group corresponding to the autonomous behavior decision model trained in the current training round belongs, and use the local decision agent in the command agent group of the opponent as a trainable agent, and use the local decision agent in the command agent group of the other opponent as an untrainable agent. Among them, the untrainable agent is deployed with an untrainable autonomous behavior decision model. By performing confrontation simulation on the trainable agent and the untrainable agent, the autonomous behavior decision model corresponding to the trainable agent is trained. Exemplarily, assuming that the current training round is to train the autonomous behavior decision model corresponding to the command agent group of the first opponent, determine the local decision agent in the command agent group of the first opponent as a trainable agent, and train the autonomous behavior decision model of the command agent group of the first opponent based on the untrainable autonomous behavior decision model deployed by the command agent group of the second opponent. Optionally, select the opponent of the trainable agent in the decision model pool of the untrainable agent based on the probability selector, and perform confrontation simulation based on the opponent and the trainable agent.

[0054] In this embodiment, the decision model pool of the untrainable agent includes one or more historical autonomous behavior decision models of the untrainable agent, wherein the historical autonomous behavior decision model is an autonomous behavior decision model obtained by training the command agent group corresponding to the untrainable agent in the historical training rounds. The probability selector is a selector that randomly selects the historical autonomous behavior decision model from the decision model pool of the untrainable agent based on a preset probability, wherein the preset probability is the pre-set selection probability of each historical autonomous behavior decision model. Exemplarily, the preset probability of the probability selector is 0.1. Each training round includes multiple confrontation simulations between the trainable agent and the opponent, wherein the opponent of each confrontation simulation is deployed with a group of historical autonomous behavior decision models. In each training round, each historical autonomous behavior decision model can be selected zero times, once, or multiple times.

[0055] Specifically, before each confrontation simulation, a group of historical autonomous behavior decision models is selected from one or more groups of historical autonomous behavior decision models in the decision model pool of the untrainable agent based on the probability selector, and the untrainable agent deployed with the selected historical autonomous behavior decision model is determined as the opponent of the trainable agent. The autonomous behavior decision model deployed in the trainable agent is trained by conducting confrontation simulation between the trainable agent and the opponent with the purpose of winning the trainable agent.

[0056] Exemplarily, after each round of confrontation simulation, the historical autonomous behavior decision model after training in the current round is added to the decision model pool, which includes historical autonomous behavior decision models trained in multiple historical confrontation simulation processes. Assuming that the decision model pool of the second adversary in the nth training round includes m groups of historical autonomous behavior decision models, the opponent of the trainable intelligent agent can select the opponent of the trainable intelligent agent from the m groups of historical autonomous behavior decision models; if the decision model pool of the second adversary in the n+2th training round includes m+1 groups of historical autonomous behavior decision models, the opponent of the trainable intelligent agent can select the opponent of the trainable intelligent agent from the m+1 groups of historical autonomous behavior decision models.

[0057] The technical solution of this embodiment selects opponents for trainable intelligent agents from a decision model pool of untrainable intelligent agents based on a probability selector. The decision model pool of each opponent includes the historical autonomous behavior decision model obtained from each round of training of the opponent, which can improve the generalization of autonomous behavior decision model training.

[0058] Specifically, when the training of the current training round meets the training end condition, the trainable agent is updated by updating the autonomous behavior decision model of the trainable agent and saving the updated autonomous behavior decision model locally in the trainable agent. The training end condition is the condition for the end of the training of the current training round. Exemplarily, the training end condition can be one or more of the current number of confrontation simulations greater than the preset number of confrontation simulations and the winning rate of the side where the trainable agent is located converges, etc. This embodiment does not limit this. Optionally, the training end condition includes that the winning rate of the side where the trainable agent is located is greater than a preset winning rate threshold.

[0059] In this embodiment, the winning rate of the trainable agent is the ratio of the number of wins of the trainable agent to the number of current confrontation simulations in the current training round w = q u,v / v, where u represents the current training round, v represents the current number of adversarial simulations, and q u,v Represents the number of wins of the agent on the side of the vth adversarial simulation in the uth training round.

[0060] Specifically, after each confrontation simulation is completed, the winning rate of the side where the trainable intelligent agent is located is calculated, and the winning rate of the side where the trainable intelligent agent is located is compared with the preset winning rate threshold, wherein the preset winning rate threshold is a preset winning rate threshold of the side where the trainable intelligent agent is located, and is used to determine the end of the training of the current training round. In the case where the winning rate of the side where the trainable intelligent agent is located is greater than or equal to the preset winning rate threshold, it can be considered that the side where the trainable intelligent agent is located can defeat the other side, and the training of the current training round is ended; in the case where the winning rate of the side where the trainable intelligent agent is located is less than the preset winning rate threshold, it can be considered that the side where the trainable intelligent agent is located cannot defeat the other side, and the training of the current training round is continued, and the next confrontation simulation is performed. Exemplarily, the preset winning rate threshold is 0.98.

[0061] The technical solution of this embodiment can ensure that the trainable intelligent agent is fully trained in the current training round by setting a preset winning rate threshold, thereby ensuring the decision accuracy of the autonomous behavior decision model of the trainable intelligent agent.

[0062] In some embodiments, the updated autonomous behavior decision model is stored in the decision model pool of the adversary party to which the trainable intelligent agent belongs, so as to enrich the decision model pool of the party to which the trainable intelligent agent belongs and improve the generalization of the autonomous behavior decision model of the other party trained in the next training round.

[0063] In some embodiments, the optional strategies of the command agent group of the second antagonist are known. Optionally, before conducting an adversarial simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, a strategy pool is called, and an adversarial simulation is conducted on the command agent group of the first antagonist based on the strategy pool, and the autonomous behavior decision model of each local decision-making agent of the command agent group of the first antagonist is trained; the strategy pool includes multiple optional strategies; accordingly, in the first adversarial simulation between the command agent group of the first antagonist and the command agent group of the second antagonist, the local decision-making agent among the command agents of the second antagonist is used as a trainable agent.

[0064] In this embodiment, the strategy pool is a pre-set strategy pool of the command intelligent agent of the second antagonist, including one or more optional strategies of the command intelligent agent of the second antagonist available to the first antagonist, wherein the optional strategies are one or more strategies set by experts in the field of multi-agent collaborative decision-making modeling technology of the second antagonist based on existing knowledge and experience, wherein each strategy includes a condition-behavior structure, for example, the structure of the strategy can be an "if, then" structure, and when the condition of the strategy is met, the strategy is triggered and the behavior is then executed.

[0065] Specifically, before the confrontation simulation between the command agent group of the first antagonist and the command agent group of the second antagonist, the local decision-making agent in the command agent group of the first antagonist is used as a trainable agent, and an optional strategy in the strategy pool of the second antagonist is randomly called in each confrontation simulation, and the trainable agent is subjected to confrontation training based on the called optional strategy. When the training end condition is met, the autonomous behavior decision model of the trainable agent is updated. In some embodiments, the updated autonomous behavior decision model of the first antagonist is stored in the decision model pool of the first antagonist. In the first training round, the agent in the command agent of the second antagonist is used as a trainable agent, and a confrontation simulation is performed with the previously trained command agent group of the first antagonist.

[0066] The technical solution of this embodiment, by calling the strategy pool, trains the autonomous behavior decision-making model of each local decision-making agent in the command agent group of the first antagonist, and in the first confrontation simulation, uses the local decision-making agent in the command agent of the second antagonist as a trainable agent, which can improve the robustness of the autonomous behavior decision-making model corresponding to the first antagonist.

[0067] In the case where the current training round is a preset training round, after the training of the autonomous behavior decision model of the current training round is completed, the training is determined to be completed. In the case where the command agent group of the first antagonist is the command agent group of one's own party, the autonomous behavior decision model corresponding to the command agent group of the first antagonist is determined as the agent decision model, wherein the agent decision model is a decision model that can be deployed in the command agent group of one's own party in actual combat to command a single-machine agent to conduct group confrontation with the opponent's agent.

[0068] For example, Figure 3 1 is a flowchart of a countermeasure simulation process provided by the first embodiment of the present invention. Figure 3 As shown, the confrontation simulation process includes:

[0069] S13001. Initialize the local decision-making agent in the command agent group of the first antagonist, and use the local decision-making agent in the command agent group of the first antagonist as a trainable agent.

[0070] S13002. Perform adversarial simulation on the trainable agent based on the strategy pool of the second adversary.

[0071] S13003. The winning rate of the trainable intelligent agent is greater than or equal to a preset winning rate threshold, and the autonomous behavior decision model updated by the trainable intelligent agent is stored in the decision model pool of the first adversary.

[0072] S13004. Determine the local decision-making agent in the command agent group of the second adversary as a trainable agent.

[0073] S13005. Based on the probability selector, a historical autonomous behavior decision model is selected from the decision model pool of the first adversary as the opponent.

[0074] S13006. The winning rate of the trainable intelligent agent is greater than or equal to a preset winning rate threshold, and the autonomous behavior decision model updated by the trainable intelligent agent is stored in the decision model pool of the second adversary.

[0075] S13007. Determine the local decision-making agent in the command agent group of the first antagonist as a trainable agent.

[0076] S13008. The winning rate of the trainable intelligent agent is greater than or equal to a preset winning rate threshold, and the autonomous behavior decision model updated by the trainable intelligent agent is stored in the decision model pool of the first adversary.

[0077] S13009. Based on the probability selector, a historical autonomous behavior decision model is selected from the decision model pool of the first adversary as the opponent.

[0078] S13010, loop S13004-S13009, and complete the training when the current training round is greater than or equal to the preset training round.

[0079] S13011. Determine the autonomous behavior decision model most recently updated by the first adversary as the intelligent agent decision model, and save the intelligent agent decision model in h5 format.

[0080] The technical solution of this embodiment is to construct a command agent group, each command agent group includes a global decision-making module and a local decision-making agent, the local decision-making agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent grouping; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; an autonomous behavior decision model of the first type of agent and an autonomous behavior decision model of the second type of agent are respectively configured; a confrontation simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation, the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist are alternately trained. After the training is completed, an agent decision model suitable for group confrontation is obtained, which can realize collaborative decision-making of multiple types of agents, and solves the problem of poor performance of large-scale global decisions by decomposing global decisions into local decisions, improves the decision accuracy of the agent decision model, and helps to improve the reference value of the decision of the command agent.

[0081] Embodiment 2

[0082] Figure 4 1 is a flowchart of a decision-making learning method for commanding an intelligent agent provided in the second embodiment of the present invention. The technical solution of the embodiment of the present invention is further optimized on the basis of any of the above embodiments. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 4 As shown, the method includes:

[0083] S210. Construct command agent groups, each command agent group includes a global decision-making module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of the first antagonist and a command agent group of the second antagonist.

[0084] S220, initializing training of the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent.

[0085] In this embodiment, the autonomous behavior decision-making model of the first type of intelligent agent and the autonomous behavior decision-making model of the second type of intelligent agent both need to be initialized and trained to ensure that the autonomous behavior decision-making model of the first type of intelligent agent and the autonomous behavior decision-making model of the second type of intelligent agent subsequently configured have autonomous decision-making capabilities.

[0086] Among them, the initialization training of any autonomous behavior decision model includes:

[0087] For local decision-making agents of the same type, the state space, action space and reward and punishment data of the autonomous behavior decision-making model in the local decision-making agent are obtained; decentralized reinforcement learning is performed on the autonomous behavior decision-making model based on the state space, action space and reward and punishment data of the autonomous behavior decision-making model and the value network in the central controller to obtain a trained autonomous behavior decision-making model, wherein the autonomous behavior decision-making models in the local decision-making agents of the same type share data during the training process.

[0088] In this embodiment, the state space is a space including one or more state information. The state information may be environment state information, state information of a local decision-making agent on one side, and at least part of the state information of a local decision-making agent on the other side. The action space is a space of optional actions of a local decision-making agent on one side, and the reward and punishment data include one or more reward data and one or more punishment data, wherein the reward data is positive feedback for a correct decision made by a local decision-making agent on one side, and the punishment is negative feedback for a wrong decision made by a local decision-making agent on one side. Based on the reward and punishment data, the decision of a local decision-making agent on one side can be gradually adjusted.

[0089] Taking the initialization training of the first type of intelligent agent as an example, the state space includes the first state information and the second state information, wherein the first state information is the battlefield state information obtained by the manned or unmanned aircraft participating in the air combat of the own side, and the battlefield state includes the observable battlefield situation information of the own side and the opponent; the second state information includes the flight state information, platform state information and load state information of the manned or unmanned aircraft of the own side and the opponent. The action space includes the first action space, the second action space and the third action space, wherein the optional actions in the first action space include the instructions to each firepower resource platform refined based on the air-to-air strike instructions sent by the global decision module; the second action space includes the strike timing and route planning determined based on the current battlefield situation; the third action space includes specific attack commands such as commanding each local decision-making intelligent agent to move forward, pause, and strike. The reward and punishment data include two reward data and two penalty data. The two reward data are respectively a positive reward of 5 for each shot down of an opponent's manned aircraft and a positive reward of 3 for each shot down of an opponent's unmanned aircraft; the two penalty data are respectively a negative penalty of 5 for the shot down of the own manned aircraft and a negative penalty of 3 for the shot down of the own unmanned aircraft.

[0090] Taking the initialization training of the second type of agent as an example, the state space includes the third state information, the fourth state information, the fifth state information and the sixth state information, wherein the third state information is the surrounding state information of the own agent formation obtained based on the global decision module, the fourth state information includes the observable state information of the other party agent obtained based on the global decision module, the fifth state information includes the position information and navigation state information of the own surface ship, and the sixth state information includes the state information of the air defense and anti-missile firepower resources of the own surface ship. The action space includes the fourth action space and the fifth action space, wherein the optional action of the fourth action space is to perform target allocation and air defense and anti-missile firepower resource selection based on the current battlefield local air attack aircraft and weapons situation, and the optional action of the fifth action space is to decide the timing and number of launching air defense firepower. The reward and punishment data include three reward data and two penalty data. The three reward data are a positive 5 reward for each opponent's manned aircraft shot down, a positive 3 reward for each opponent's unmanned aircraft shot down, and a positive 2 reward for each opponent's anti-ship weapon shot down; the two penalty data are a negative 10 penalty for one's own ship being hit, and a negative 20 penalty for one's own aircraft carrier being hit.

[0091] The central controller is a controller used to assist the centralized training of multiple local decision-making agents corresponding to any autonomous behavior decision-making model during the centralized training process. Decentralized reinforcement learning includes centralized training and decentralized decision-making.

[0092] Specifically, in the centralized training process of any autonomous behavior decision model, the initialized value network is deployed to the central controller, and the initialized policy network is deployed to each local decision agent. The policy network is the autonomous behavior decision model to be initialized. Each local decision agent collects experience data by interacting with the environment and stores the data in a shared experience pool so that other local decision agents can access and learn. The environment includes the real-time state space of agent u. and action space Multiple local decision-making agents can be trained in a distributed manner.

[0093] For example, Figure 5 : is a flow chart of an autonomous behavior decision model training process provided by the second embodiment of the present invention. Figure 5 As shown, the autonomous behavior decision model training process includes:

[0094] S22001, initialize the evaluation network and the policy network, and randomly initialize the neural network; the initialized evaluation network is expressed as The policy network is denoted as π u, the parameters of the neural network are represented as w, and the hyperparameters of the neural network are represented as θ, so as to optimize w and θ through training.

[0095] S22002: Initialize the playback cache; wherein the playback cache is represented by D.

[0096] S22003. Within one action step, each local decision agent executes the strategy function to generate an action; wherein the strategy function corresponding to the local decision agent u is expressed as

[0097] S22004, based on the reward and punishment data and the simulation environment, the reward and punishment value and the next state space are obtained; wherein the reward and punishment value is represented by r t , the next state space is represented by s t+1 .

[0098] S22005. Repeat S22003 and S22004 until the end of a game.

[0099] S22006. Obtain the trajectory of each local decision-making agent in the game, and calculate the value and advantage; where the trajectory is expressed as T is the trajectory length, and the value is expressed as The advantage is expressed as

[0100] S22007, storing the relevant data of the game into the playback buffer; wherein the relevant data of the game is represented as

[0101] S22008. Disrupt the order of related data in the playback cache and renumber them.

[0102] S22009. Set the sampling times to zero; wherein the sampling times is represented by j.

[0103] S22010, sampling a batch of relevant data from the playback buffer; wherein the batch size is represented by B, and the sampled relevant data is represented by

[0104] S22011, traverse each local decision-making agent, obtain the gradient, and update the gradient based on the Adam gradient descent method, and increase the number of sampling times by 1; where θ u The gradient is Δθ u , w u The gradient is Δw u , updated Δθ u and Δw u as follows:

[0105]

[0106]

[0107] S22012. Loop S22010 and S22012 until j>T / B.

[0108] S22013, loop S22008-S22013 until the set number of steps is reached.

[0109] S22014, loop S22002-S22014, iterate to the set number of rounds.

[0110] S22015. Save the strategy model and complete the training.

[0111] S230, respectively configure an autonomous behavior decision model for the first type of intelligent agent and an autonomous behavior decision model for the second type of intelligent agent.

[0112] S240, conduct confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation, alternately train the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist, and after the training is completed, obtain an agent decision model suitable for group confrontation.

[0113] The technical solution of this embodiment is to construct a command agent group, each command agent group includes a global decision-making module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; the autonomous behavior decision model of the first type of agent and the autonomous behavior decision model of the second type of agent are initialized and trained; the autonomous behavior decision model of the first type of agent and the autonomous behavior decision model of the second type of agent are respectively configured; the command agent group of the first antagonist and the command agent group of the second antagonist are subjected to confrontation simulation, and during the confrontation simulation process, the autonomous behavior decision model of each local decision agent respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist is alternately trained, and after the training is completed, an agent decision model suitable for group confrontation is obtained, which can ensure that the autonomous behavior decision model configured to the first type of agent and the autonomous behavior decision model of the second type of agent have autonomous decision-making capabilities, thereby improving the training efficiency of the confrontation simulation of the command agent group of the first antagonist and the command agent group of the second antagonist.

[0114] Embodiment 3

[0115] Figure 6 Schematic diagram of a decision-making learning device for commanding an intelligent agent provided in Embodiment 3 of the present invention. Figure 6As shown, the device comprises:

[0116] A command agent group construction module 310 is used to construct a command agent group, each command agent group includes a global decision module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of the first antagonist and a command agent group of the second antagonist;

[0117] An autonomous behavior decision model configuration module 320, for respectively configuring an autonomous behavior decision model of a first type of intelligent agent and an autonomous behavior decision model of a second type of intelligent agent;

[0118] The confrontation simulation module 330 is used to perform confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation process, alternately train the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist, and after the training is completed, obtain an agent decision model suitable for group confrontation.

[0119] The technical solution of this embodiment is to construct a command agent group, each command agent group includes a global decision-making module and a local decision-making agent, the local decision-making agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent grouping; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; an autonomous behavior decision model of the first type of agent and an autonomous behavior decision model of the second type of agent are respectively configured; a confrontation simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation, the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist are alternately trained. After the training is completed, an agent decision model suitable for group confrontation is obtained, which can realize collaborative decision-making of multiple types of agents, and solves the problem of poor performance of large-scale global decisions by decomposing global decisions into local decisions, improves the decision accuracy of the agent decision model, and helps to improve the reference value of the decision of the command agent.

[0120] Based on the above embodiment, optionally, a global decision module is created based on expert rules to compare the target distance of the combat object with the preset distance to determine the type of command grouping task; the types of grouping tasks include first type and second type; the first type of intelligent agent is used to perform the first type of grouping task, and the second type of intelligent agent is used to perform the second type of grouping task.

[0121] On the basis of the above embodiments, optionally, the first type of intelligent agent is a long-range strike formation command intelligent agent, and the second type of intelligent agent is a medium- and short-range strike formation command intelligent agent.

[0122] On the basis of the above-mentioned embodiment, optionally, the confrontation simulation module 330 is specifically used to: determine the local decision-making agent in the command agent group of the second antagonist or the command agent group of the first antagonist as the trainable agent in an alternating manner, perform confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the simulation process, train the autonomous behavior decision model of the trainable agent until the training end condition is met and the trainable agent is updated.

[0123] On the basis of the above-mentioned embodiment, optionally, the decision-making learning device of the command intelligent agent also includes a strategy pool calling module, which is used to call the strategy pool before the confrontation simulation module 330, perform confrontation simulation on the command intelligent agent group of the first antagonist based on the strategy pool, and train the autonomous behavior decision model of each local decision intelligent agent of the command intelligent agent group of the first antagonist; the strategy pool includes multiple optional strategies; accordingly, in the first confrontation simulation between the command intelligent agent group of the first antagonist and the command intelligent agent group of the second antagonist, the intelligent agents in the command intelligent agent of the second antagonist are used as trainable intelligent agents.

[0124] On the basis of the above embodiment, optionally, the adversarial simulation module 330 is also used to: select an opponent for the trainable agent from the decision model pool of the untrainable agent based on a probability selector, and perform adversarial simulation between the opponent and the trainable agent; and the training end condition includes that the winning rate of the side of the trainable agent is greater than a preset winning rate threshold.

[0125] On the basis of the above embodiment, optionally, the decision-making learning device of the command intelligent agent also includes an autonomous behavior decision model initialization training module, which is used to initialize the training of the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent before the autonomous behavior decision model configuration module 320, wherein the initialization training of any autonomous behavior decision model includes: for local decision-making agents of the same type, obtaining the state space, action space and reward and punishment data of the autonomous behavior decision model in the local decision-making agent; performing decentralized reinforcement learning on the autonomous behavior decision model based on the state space, action space and reward and punishment data of the autonomous behavior decision model and the value network in the central controller to obtain a trained autonomous behavior decision model, wherein the autonomous behavior decision models in the local decision-making agents of the same type share data during the training process.

[0126] The decision-making learning device for the command intelligent agent provided in the embodiment of the present invention can execute the decision-making learning method for the command intelligent agent provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0127] Embodiment 4

[0128] Figure 7 1 is a schematic diagram of the structure of an electronic device for implementing the decision-making learning method of the command intelligent agent of an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0129] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0131] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the decision-making learning method of commanding an intelligent agent.

[0132] In some embodiments, the decision-making learning method of the command agent may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the decision-making learning method of the command agent described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the decision-making learning method of the command agent in any other appropriate manner (e.g., by means of firmware).

[0133] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The computer program for implementing the decision-making learning method of the command intelligent agent of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

[0135] Embodiment 5

[0136] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a decision-making learning method for commanding an intelligent agent, the method comprising:

[0137] Construct command agent groups, each command agent group includes a global decision module and a local decision agent, the local decision agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent grouping; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; configure an autonomous behavior decision model of the first type of agent and an autonomous behavior decision model of the second type of agent respectively; conduct confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation, alternately train the autonomous behavior decision models of the local decision agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist, and after the training is completed, obtain an agent decision model suitable for group confrontation.

[0138] In the context of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. More specific examples of machine readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein may be implemented in a computing system that includes a backend (e.g., as a data server), or a computing system that includes middleware (e.g., an application server), or a computing system that includes a frontend (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backends, middleware, or frontends. The systems may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0141] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A decision-making learning method for commanding an intelligent agent, characterized in that: include: Construct a command agent group, each of which includes a global decision-making module and a local decision-making agent, the local decision-making agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; the global decision-making module is a module for making global decisions, and plays the role of a formation commander in the command agent group; wherein the global decision is a decision on a single agent to perform a global task, including dividing the agent group and planning the attack task of the agent group; the local decision-making agent is a command agent for making local decisions, and plays the role of a formation commander in the command agent group. The role of the commander; wherein the local decision is a decision on the execution of a local task by a single-machine intelligent agent, including planning a single-machine intelligent agent strike task; the global decision module is created based on expert rules; the conditions of the expert rules include at least part of the target distance, the number of targets and the current firepower resources, wherein the target distance is the distance between the single-machine intelligent agent of the combat object and the position of the own party, the number of targets is the number of single-machine intelligent agents of the combat object, and the current firepower resources are the firepower resources currently available to the own party; the intelligent agent group is an air strike group command intelligent agent or a reconnaissance and strike mixed intelligent agent group; the single-machine intelligent agent in the air strike group command intelligent agent is an aircraft; the single-machine intelligent agent in the reconnaissance and strike mixed intelligent agent group is a surface ship; Respectively configuring the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent, so that each type of intelligent agent can autonomously execute the corresponding type of grouping task sent to the type of intelligent agent by the global decision module; A confrontation simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist. During the confrontation simulation, the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist are alternately trained. After the training is completed, an agent decision model suitable for group confrontation is obtained.

2. The method according to claim 1, characterized in that The global decision module is used to compare the target distance of the combat object with the preset distance to determine the type of command grouping task; the type of grouping task includes a first type and a second type; The first type of agent is used to perform the first type of grouping task, and the second type of agent is used to perform the second type of grouping task.

3. The method according to claim 2, characterized in that The first type of intelligent agent is a long-range strike formation command intelligent agent, and the second type of intelligent agent is a medium- and short-range strike formation command intelligent agent.

4. The method according to claim 1, characterized in that: The performing confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist comprises: The command agent group of the second antagonist or the local decision-making agent in the command agent group of the first antagonist is determined in an alternating manner as a trainable agent, and an adversarial simulation is performed on the command agent group of the first antagonist and the command agent group of the second antagonist. During the simulation, the autonomous behavior decision model of the trainable agent is trained until the training end condition is met and the trainable agent is updated.

5. The method according to claim 1, characterized in that: Before performing confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, the method further includes: Calling a strategy pool, performing confrontation simulation on the command agent group of the first antagonist based on the strategy pool, and training the autonomous behavior decision model of each local decision agent of the command agent group of the first antagonist; the strategy pool includes a plurality of optional strategies; Correspondingly, in the first confrontation simulation between the command agent group of the first antagonist and the command agent group of the second antagonist, the local decision-making agent in the command agent of the second antagonist is used as a trainable agent.

6. The method according to claim 4, characterized in that The performing confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist comprises: Selecting a battle object for the trainable agent from a decision model pool of untrainable agents based on a probability selector, and performing a confrontation simulation between the battle object and the trainable agent; as well as, The training end condition includes that the winning rate of the side to which the trainable intelligent agent belongs is greater than a preset winning rate threshold.

7. The method according to claim 1, characterized in that Before respectively configuring the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent, the method further includes: Initializing training of the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent, wherein the initialization training of any of the autonomous behavior decision models includes: For the local decision-making agents of the same type, obtaining the state space, action space and reward and punishment data of the autonomous behavior decision model in the local decision-making agents; Based on the state space, action space and reward and punishment data of the autonomous behavior decision-making model and the value network in the central controller, decentralized reinforcement learning is performed on the autonomous behavior decision-making model to obtain a trained autonomous behavior decision-making model, wherein the autonomous behavior decision-making models in the local decision-making agents of the same type share data during the training process.

8. A decision-making learning device for commanding an intelligent agent, characterized in that: include: A command agent group construction module is used to construct a command agent group, each of which includes a global decision-making module and a local decision-making agent, the local decision-making agent includes a first type of agent and a second type of agent, and each type of agent includes at least one agent group; the command agent group includes a command agent group of a first antagonist and a command agent group of a second antagonist; the global decision-making module is a module for making global decisions, and plays the role of a formation commander in the command agent group; wherein the global decision is a decision on a single agent to perform a global task, including dividing the agent group and planning the agent group's strike task; the local decision-making agent is a command agent for making local decisions, and in the command agent The local decision is a decision on the execution of local tasks by a single-machine intelligent agent, including planning the strike tasks of a single-machine intelligent agent; the global decision module is created based on expert rules; the conditions of the expert rules include at least part of the target distance, the number of targets and the current firepower resources, wherein the target distance is the distance between the single-machine intelligent agent of the combat object and the position of the own party, the number of targets is the number of single-machine intelligent agents of the combat object, and the current firepower resources are the firepower resources currently available to the own party; the intelligent agent formation is an air strike formation command intelligent agent or a reconnaissance and strike mixed intelligent agent formation; the single-machine intelligent agent in the air strike formation command intelligent agent is an aircraft; the single-machine intelligent agent in the reconnaissance and strike mixed intelligent agent formation is a surface ship; An autonomous behavior decision model configuration module, used to configure the autonomous behavior decision model of the first type of intelligent agent and the autonomous behavior decision model of the second type of intelligent agent, respectively, so that each type of intelligent agent can autonomously execute the corresponding type of grouping task sent to the type of intelligent agent by the global decision module; The confrontation simulation module is used to perform confrontation simulation on the command agent group of the first antagonist and the command agent group of the second antagonist, and during the confrontation simulation process, alternately train the autonomous behavior decision models of the local decision-making agents respectively included in the command agent group of the first antagonist and the command agent group of the second antagonist, and after completing the training, obtain an agent decision model suitable for group confrontation.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the decision-making learning method for commanding an intelligent agent according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the decision-making learning method for a command intelligent agent according to any one of claims 1 to 7 when executed.

Images