General decision-making architecture and method for time-varying network

Abstract

The invention discloses a general decision-making architecture and method for a time-varying network, and mainly solves the problems that the prior art cannot adapt to real-time changes in an adversarial environment and cannot generate tasks consistent with those of an upper level autonomously under the condition that the network is disconnected. According to the implementation scheme, a command center takes an environment state as input, and generates subtasks under the constraint of an overall confrontation target; the command center distributes the subtasks to the confrontation units by using deep reinforcement learning; the confrontation unit determines subtasks to be completed according to the communication condition with the command center; the confrontation unit executes the determined subtask. The system can be compatible with a centralized decision-making architecture and an autonomous decision-making architecture at the same time, and can perform adaptive switching according to a network connection condition, so that the confrontation units can support subtasks in the same group or autonomously generate subtasks when being disconnected with a command center; and the consistency of the subtasks generated by the autonomous decision and the superior allocation tasks can be ensured by calculating the similarity of the task action vectors, and the method can be used for an electronic countermeasure system.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a network decision architecture and method, which can be used in an electronic countermeasure system. Background Art

[0002] With the continuous development of artificial intelligence technology, modern confrontation forms are moving towards networking, intelligence and unmanned. Unmanned confrontation systems are increasingly used in confrontation, and in complex and dynamically changing confrontation environments, it is particularly necessary to combine manned and unmanned confrontation modes. Manned / unmanned collaborative confrontation is a confrontation method in which manned confrontation platforms and unmanned confrontation platforms are jointly organized to jointly implement coordinated attacks in an informationized, networked and system confrontation environment. Confrontation task planning is the core of the decision-making information system, which is responsible for generating resource allocation and target strike plans based on the confrontation target, the opponent's confrontation array network, and confrontation resources and rules to maximize the benefits of task execution at this level. However, the current task planning system is mainly applicable to pre-set situations, and its ability to respond to unpredictable events is relatively weak. Collaborative task allocation involves more complex problems, and its solution space grows exponentially with the increase in the number and scale of participating systems, making it an NP-hard problem with multiple decision variables and multiple constraints. Solving these problems requires more advanced and complex algorithms and systems to adapt to real-time changes and uncertainties in the confrontation environment.

[0003] The patent document with application number 202210643334.5 discloses an intelligent communication architecture and method, which includes a communication requirement module, a decision criterion module, a decision space module, a knowledge base module, an intelligent perception module, a communication resource decision module, an intelligent communication solution module and a sample update module. The communication requirement module, the decision criterion module and the decision space module are set before communication in a complex electromagnetic environment; the intelligent perception module includes a search decision unit, an intelligent decision unit and a decision correction unit; the search decision unit is used to search the knowledge base; the intelligent decision unit is used to make intelligent decisions when there is no sample in the knowledge base that is the same as the dynamic environment information or the knowledge base is too large; the decision correction unit is used to make corrections when the intelligent communication performance feedback of the intelligent communication solution module is obtained; the communication resource decision module includes system parameter decision and spectrum resource decision; the intelligent communication solution module communicates according to the resource decision result. When the intelligent perception module receives dynamic environmental information, it uses the search decision unit to search the knowledge base. If there are samples with the same environmental information in the knowledge base, the existing samples are directly used to configure the communication link parameters. If the samples do not exist or the knowledge base is too large, the intelligent decision unit is used to make decisions. After the decision result is obtained, the parameters are configured and communication is carried out through the intelligent communication module. At the same time, based on the communication performance feedback, if the feedback result does not meet the communication requirements, the decision correction unit is used to make a new decision until the feedback meets the communication requirements, and the decision result is stored in the knowledge base in the form of a sample. Since this architecture is mainly configured and optimized according to various communication conditions to meet the communication performance required for decision-making; therefore, when the connection with the superior is completely lost, and when the communication conditions frequently switch on and off, the architecture cannot work properly.

[0004] The patent document with application number 202211168943.6 discloses an intelligent decision-making method and architecture for a drone communication system, which includes a decision analysis layer, a task planning layer, an algorithm design and collaborative control (ADCC) layer, and a task execution layer. The task execution layer is used for the drone to interact with the environment and perceive environmental status information; the ADCC layer sends the environmental status information obtained by the task execution layer and the current drone information to the task planning layer; the task planning layer calculates the system dynamic reachability performance based on the drone information and environmental information received from the ADCC layer, and transmits the calculated performance, environmental information, and drone information to the decision analysis layer; the decision analysis layer receives the current system dynamic reachability performance, environmental information, and drone information sent by the task planning layer, and judges whether the performance indicators of the current drone resource management decision meet the preset goals according to the resource management tasks. If not, it is optimized; the optimization process is: the decision analysis layer establishes the optimization problem according to the set task goals and determines the constraints; the task planning layer formulates solutions; the ADCC layer designs the processing method of the solution to obtain the drone control strategy; the task execution layer maps the drone strategy to physical behavior and then interacts with the environment. This architecture mainly makes optimization decisions on the drone network communication mode according to various environmental conditions to achieve a stable and efficient communication mode. In this architecture, it is assumed that there are multiple optional network communication methods, and the main goal of the decision is the drone communication itself, not the confrontation. Therefore, it cannot be used for confrontation involving multiple units, nor can the subordinates autonomously generate tasks and maintain consistency with the superiors when the network is disconnected. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a general decision-making architecture and method for time-varying networks, so as to dynamically adjust the multi-agent decision-making architecture according to the network communication situation in an adversarial environment, support decision-making under two network communication states: normal communication and loss of connection with the superior; and ensure that when the connection with the superior is lost, the subordinate adversarial unit can independently decide to generate subtasks, and the generated subtasks are consistent with the decisions of the superior command center.

[0006] To achieve the above object, the technical solution of the present invention includes the following:

[0007] 1. A general decision-making architecture for time-varying networks, running on a superior command center and several adversarial units, characterized by:

[0008] The decision architecture selection module is used to periodically maintain the communication table information in the environmental observation value and select the decision mode according to the disconnection relationship between the intelligent agent and the command center;

[0009] The subtask generation module is used to generate subtasks based on the agent's environmental observations under the constraints of the overall adversarial goal, and to maintain the consistency of the lower-level subtasks with the upper-level subtasks when generating subtasks;

[0010] An adversarial unit assignment module, used to assign the subtasks generated by the subtask generation module to each adversarial unit using deep reinforcement learning;

[0011] The subtask execution module is used to execute the subtasks output by the adversarial unit allocation module, and update and push local environment observations to neighbors based on the execution results;

[0012] The autonomous decision-making module is used to make local autonomous decisions when the confrontation unit is disconnected from the superior command center, that is, to determine the tasks that the local confrontation unit needs to perform.

[0013] 2. A general decision-making method for time-varying networks, characterized in that it comprises the following steps:

[0014] 1) The command center uses the subtask generation module to take the environment state as input and generate subtasks under the constraints of the overall confrontation goal to maintain the consistency of the subtasks and the overall confrontation goal;

[0015] 2) The command center uses deep reinforcement learning to assign subtasks to each adversary unit through the adversary unit assignment module;

[0016] 3) Each adversarial unit queries the communication table through the decision architecture selection module before making a decision to determine whether the agent is disconnected from the command center:

[0017] If there is no disconnection, agent i sends its partial observation value ob to the command center i , the command center receives some observation values ​​ob i Then, step 1) is executed periodically to make a centralized decision and assign subtasks to the agents corresponding to the received observations;

[0018] Otherwise, the agent on the lower-level confrontation unit makes an autonomous decision through the autonomous decision-making module and executes step 4);

[0019] 4) The adversarial unit determines whether there are any unfinished subtasks:

[0020] If there are unfinished local subtasks, continue to complete the subtasks assigned by the command center through the confrontation unit allocation module;

[0021] If the local subtask has been completed, but other subtasks in the original group have not been completed, other subtasks will be supported through the contract network protocol;

[0022] If both the local and original grouping subtasks have been completed, a new subtask that is consistent with the superior decision is generated autonomously through the subtask generation module;

[0023] 5) The subtask execution module implements the subtasks determined in step 4), updates the local environment observations and pushes them to neighbors.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] First, the present invention proposes a general decision-making architecture for time-varying networks, and designs a decision-making architecture selection module to determine whether the adversarial unit needs to make an autonomous decision. Compared with the existing solutions, the present invention is compatible with both the centralized decision-making architecture and the autonomous decision-making architecture, giving priority to the centralized decision of the superior, and can switch adaptively according to the network connectivity.

[0026] Secondly, the present invention proposes a general decision-making method for time-varying networks and trains deep reinforcement learning networks to generate subtasks; compared with existing solutions, the present invention enables the adversarial unit to support other subtasks in the same group or generate subtasks through autonomous decision-making when it is disconnected from the command center, and ensures the consistency of the subtasks generated by autonomous decision-making with the tasks assigned by superiors by calculating the similarity of task action vectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a framework diagram of the present invention;

[0028] Figure 2 It is a general flow chart for realizing the method of the present invention; DETAILED DESCRIPTION

[0029] The embodiments of the present invention are described in detail below with reference to the accompanying drawings:

[0030] Reference Figure 1 ,This example is a general decision-making architecture for time-varying networks, which includes a decision-making architecture selection module, a subtask generation module, an adversarial unit allocation module, a subtask execution module, and an autonomous decision-making module, among which:

[0031] The decision architecture selection module is used to periodically maintain the communication table information in the environmental observation value, establish an adjacency matrix based on the direct connection relationship, use the depth-first algorithm to determine the connection between the adversarial unit and the command center, and select the decision method based on the disconnection relationship between the intelligent agent and the command center;

[0032] The subtask generation module is used to generate subtasks under the constraints of the overall adversarial goal using a deep reinforcement learning network based on the agent's environmental observations. When autonomously generating subtasks in the autonomous decision-making mode, the similarity of the action vectors of the lower-level subtask and the upper-level subtask is incorporated into the reward value calculation to maintain the consistency of the lower-level subtask and the upper-level subtask.

[0033] An adversarial unit allocation module is used to allocate the subtasks generated by the subtask generation module to each adversarial unit using a deep reinforcement learning network and form task groups;

[0034] The subtask execution module is used to execute the subtasks output by the adversarial unit allocation module, and update and push local environment observations to neighbors based on the execution results;

[0035] The autonomous decision-making module is used to make local autonomous decisions when the confrontation unit is disconnected from the superior command center, that is, to determine the tasks that the local confrontation unit needs to perform:

[0036] If its own subtask is not completed, complete its own subtask first;

[0037] If there are unfinished subtasks in the same group, support the subtasks in the same group first;

[0038] If all subtasks in the same group have been completed, the subtask generation module will be used to generate subtasks autonomously. Figure 2 , the general decision-making method for time-varying networks in this example includes the following steps:

[0039] Step 1: The command center generates subtasks.

[0040] The command center uses the subtask generation module to take the environment state as input and generate subtasks under the constraints of the overall confrontation goal to maintain the consistency of the subtasks and the overall confrontation goal. The specific implementation includes the following:

[0041] 1.1) For the reinforcement learning strategy network GN1=π to be used for subtask generation θ To train:

[0042] 1.1.1) Set the type of units involved in the confrontation between the two sides at time t to type t 、Number count t 、location t The feature set s t ={(type t , count t ,location t )} and the overall confrontation target G total are all input into the reinforcement learning strategy network, and the overall task action T is output action , task location T location , Mission Objective T target , and task time T time ;

[0043] 1.1.2) Adopt a heuristic subtask generation strategy to generate the initial subtask set through expert knowledge in is the i-th initial subtask including attack and defense, 0≤i≤n, n is the total number of subtasks;

[0044] 1.1.3) Initialize the reinforcement learning strategy network parameters θ;

[0045] 1.1.4) Randomly generate subtasks T from the initial subtask set i The collection of local subtask space T local :

[0046]

[0047]

[0048] Among them, δ is the preset local adjustment range, Δ i is a random mutation vector, distance(·) represents T i and The Euclidean distance between these two vectors;

[0049] 1.1.5) According to the generated subtask set T = {T1, T2, ..., T i , ..., T n} and the current state s t Calculate the subtask set reward score Score(T, s t ):

[0050]

[0051] Among them, w i For task T i The weight of Task(T i ,s t ) is task T i score;

[0052] 1.1.6) Update the reinforcement learning strategy network parameters θ by gradient descent based on the reward value score:

[0053]

[0054] Among them, α is the learning rate that determines the size of the parameter update step. represents the gradient of parameter θ, and E[·] represents the expectation;

[0055] 1.1.7) Repeat steps 1a4) to 1a6) until the reinforcement learning policy network converges to obtain a trained reinforcement learning policy network;

[0056] 1.2) Use the trained reinforcement learning policy network for reasoning and generate subtasks under the overall adversarial goal constraints:

[0057] 1.2.1) Set the type of units involved in the confrontation between the two sides at time t1 to type t1 、Number count t1 、location t1 Composition feature set t1 :

[0058] s t1 ={(type t1 , count t1 ,location t1 )}

[0059] 1.2.2) s t1 and the overall confrontation target G total Input into the trained reinforcement learning strategy network, and output the subtask set T at time t1 that meets the overall goal t1 :

[0060]

[0061] in, is the i-th subtask at time t1, including task actions Mission Location Mission Objectives and task time

[0062] Step 2: The command center assigns subtasks to the confrontation unit.

[0063] The command center uses deep reinforcement learning to assign subtasks to each adversary unit through the adversary unit assignment module. The specific implementation includes the following:

[0064] 2.1) Training the reinforcement learning policy network to be used for adversarial unit allocation:

[0065] 2.1.1) Initialize the existing reinforcement learning policy network using random initialization, Xavier initialization, and He initialization methods Parameters θ1 and evaluation network parameters φ;

[0066] 2.1.2) Select the existing target network Q′ φ and Q Network φ , and the target network Q′ φ The parameters of the Q network are initialized to φ The parameters are the same as:

[0067] Q′ φ ←Q φ

[0068] 2.1.3) The environmental state s t, subtask set {T1, T2, ..., T n} and the set of available adversarial units {U1, U2, ..., U m} Input reinforcement learning policy network π θ1 , output action a t :

[0069] a t ={group j ((T j1 , U ja ), (T j2 , U jb ),...,(T jk , U jx ),...,(T jn , U jy ))}

[0070] Among them, group j (·) represents the combination of the jth group of cooperative confrontation units, (T jk , U jx ) means that in the jth group, the number U jx The adversarial unit is assigned to T jk subtasks;

[0071] 2.1.4) Update each assigned subtask T according to the assignment result i The status is completed, and the environment status s is updated according to the task actions, task locations, task goals, and task times of these newly completed subtasks. t+1 ;

[0072] 2.1.5) Calculate the target Q value y of the reinforcement learning strategy network t :

[0073]

[0074] Among them, the target Q value y t is the future discounted reward the agent expects to receive after taking an action; r t is the immediate reward obtained by the agent from the environment at time t; γ is a discount factor used to weigh the importance of immediate rewards and future rewards; is in state s t+1 Next, according to the current reinforcement learning strategy network The Q value corresponding to the selected action;

[0075] 2.1.6) Calculate the predicted Q value of the evaluation network

[0076]

[0077] Among them, Q φ (s t , a t ) is the evaluation network with parameter φ in state s t and action a t The Q value output under

[0078] 2.1.7) Calculate the loss function L used to update the Q network through the back-propagation algorithm Q (φ):

[0079]

[0080] 2.1.8) Minimize the loss function through the gradient descent algorithm and update the parameter φ of the evaluation network so that its predicted Q value is closer to the target Q value:

[0081]

[0082] Where α is the learning rate that determines the step size of parameter update, is the loss function L Q (φ) Gradient with respect to parameter φ;

[0083] 2.1.9) Computing the reinforcement learning policy network The loss function L π (θ1):

[0084] L π (θ)=-Q φ (s t , π θ (s t ))

[0085] in, For state s t According to the reinforcement learning policy network The Q value corresponding to the selected action;

[0086] 2.1.10) Calculate the loss function L π (θ1) Gradient of the reinforcement learning policy network parameter θ1 And update the parameter θ1:

[0087]

[0088] 2.1.11) Update the target network parameter φ′ by the following formula:

[0089] φ′←τφ+(1-τ)φ′0<τ<1

[0090] Among them, τ is the soft update coefficient;

[0091] 2.1.12) Repeat steps 2.1.4) to 2.1.11) until the reinforcement learning policy network Converge and obtain a trained reinforcement learning strategy network;

[0092] 2.2) The environmental state s at time t2 t2 , subtask collection and the set of available adversarial units {U1, U2, ..., U m} Input the trained reinforcement learning strategy network Output action a t2 :

[0093]

[0094] in, Indicates that at time t2, the number U jx The opposing unit is assigned to subtasks.

[0095] Step 3: The confrontation unit determines the subtask to be completed based on the connection status with the command center.

[0096] 3.1) Each adversarial unit queries the communication table through the decision architecture selection module before making a decision to determine whether the intelligent agent is disconnected from the command center. The implementation steps include the following:

[0097] 3.1.1) Each agent i observes the environment according to period T0 and obtains the local observation value O i ;

[0098] 3.1.2) Each agent i that performs a task creates a communication table to record the network direct connection between itself and other neighboring agents j Direct connection to the command center and hop count TTL i ; If agent i is directly connected to agent j, then otherwise Similarly, if agent i is directly connected to the command center, then otherwise

[0099] 3.1.3) All agents follow the cycle T com Exchange the communication table information with all neighbors in the communication table to obtain the direct adjacency matrix M of the entire system:

[0100]

[0101] 3.1.4) Each agent i queries the matrix M and performs a depth-first search:

[0102] For each neighbor agent j, if Then, we further recursively search all reachable neighbors k of agent j according to matrix M, and set Repeat this recursive process until all directly connected agents and their neighbors have been traversed;

[0103] examine If the value of It means that agent i can directly or indirectly connect to the command center and execute step 3.3);

[0104] on the contrary It means that agent i is disconnected from the command center, and execute step 3.4);

[0105] 3.3) Agent i sends its partial observation value ob to the command center i , the command center receives some observation values ​​ob i Then, step 1 is periodically executed to make a centralized decision and assign subtasks to the agents corresponding to the received observations;

[0106] 3.4) The agent on the lower-level adversarial unit makes an autonomous decision through the autonomous decision-making module and executes step 3.5);

[0107] 3.5) Determine whether there are unfinished subtasks in the local and confrontation unit groups, and determine the tasks to be performed:

[0108] If there are unfinished local subtasks, execute step 4) and continue to execute the subtasks assigned by the command center through the confrontation unit assignment module;

[0109] If the local subtask has been completed and other subtasks of the original group have not been completed, execute step 3.6);

[0110] If both the local and original grouping subtasks have been completed, proceed to step 3.7);

[0111] 3.6) Support other subtasks through the contract network protocol:

[0112] 3.6.1) The opposing unit with the unfinished subtask broadcasts the task requirements, including capabilities and location information, and the opposing unit decides whether to accept based on its own capabilities;

[0113] 3.6.2) Select the corresponding opposing units that received the message first among the opposing units that accepted the task, and send them the task confirmation message;

[0114] 3.6.3) Add the selected opposing unit to its own sub-environment to perform the task;

[0115] 3.7) Execute the subtask generation module on the adversarial unit to autonomously generate new subtasks:

[0116] 3.7.1) On the adversarial unit, the existing reinforcement learning strategy network GN3 = π is used for autonomous decision making. θ3 To train:

[0117] 3.7.1a) Set the type of units involved in the confrontation between the two sides at time t to type t 、Number count t 、location t The feature set s t ={(type t , count t ,location t )} and the overall confrontation target G total are all input into the reinforcement learning strategy network, and the overall task action T is output action , task location T location , Mission Objective T target , and task time T time ;

[0118] 3.7.1b) Adopt a heuristic subtask generation strategy to generate the initial subtask set through expert knowledge in is the i-th initial subtask including attack and defense, 0≤i≤n, n is the total number of subtasks;

[0119] 3.7.1c) Initialize the reinforcement learning strategy network parameters θ3;

[0120] 3.7.1d) Randomly generate subtasks T from the initial subtask set i The collection of local subtask space T local :

[0121]

[0122]

[0123] Among them, δ is the preset local adjustment range, Δ i is a random mutation vector,

[0124] distance(·) represents T i and The Euclidean distance between these two vectors;

[0125] 3.7.1e) According to the generated subtask set T = {T1, T2, ..., T i , ..., T n} and the current state s tCalculate the reward value score S1(T, s t ):

[0126]

[0127] Among them, w i For task T i The weight of Task(T i ,s t ) is task T i Score, AT i To autonomously generate subtask T i The corresponding action vector; AT i 0 When the connection is not disconnected before the autonomous generation of the subtask, the action vector corresponding to the subtask assigned by the superior to this adversarial unit, Cosine_Similarity(AT i , AT i 0 ) is the cosine similarity between the action vectors corresponding to the subtask generated by the self-decision after disconnection from the superior and the subtask assigned by the nearest superior, and its calculation formula is:

[0128]

[0129] Where A and B are the two vectors to be compared, AB represents the dot product of the vectors, also known as the inner product, which is the sum of the products of the corresponding elements of the two vectors; ||A|| and ||B|| represent the modulus of the vectors, which can be obtained by calculating the sum of the squares of each vector element and then taking the square root;

[0130] 3.7.1f) Update the reinforcement learning policy network parameters θ3 by gradient descent based on the reward value score:

[0131]

[0132] Where α is the learning rate used to determine the step size of parameter updates. represents the gradient of parameter θ3, and E[·] represents the expectation;

[0133] 3.7.1g) Repeat steps 3.7.1d) to 3.7.1f) until the reinforcement learning policy network converges to obtain a trained reinforcement learning policy network GN3 for autonomous decision-making;

[0134] 3.7.2) Perform the reasoning phase on the adversarial unit and output a set of subtasks that meet the current overall goal based on the observation of the other party:

[0135] 3.7.2a) Set the type of units involved in the confrontation at time t3 to type t3 、Number countt3 、location t3 Composition feature set t3 :

[0136] s t3 ={(type t3 , count t3 ,location t3 )}

[0137] 3.7.2b) s t3 and the overall confrontation target G total Input into the trained reinforcement learning strategy network, and output the subtask set T at time t3 that meets the overall goal t3 :

[0138]

[0139] in, is the i-th subtask at time t3, including task actions Mission Location Mission Objectives and task time

[0140] Step 4: The adversarial unit performs the determined subtasks.

[0141] 4.1) Each assigned task adversarial unit i performs an action and updates the environment observation o i ;

[0142] 4.2) After decision making and execution, agent i pushes its local observation value o to the neighboring nodes in the communication table. i and the current time.

[0143] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

[0144] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the sequence of the step numbers is not limited.

Claims

1. A general decision-making architecture for time-varying networks, running on a superior command center and several adversarial units, characterized by: include: The decision architecture selection module is used to periodically maintain the communication table information in the environmental observation value and select the decision mode according to the disconnection relationship between the intelligent agent and the command center; The subtask generation module is used to generate subtasks based on the agent's environmental observations under the constraints of the overall adversarial goal, and to maintain the consistency of the lower-level subtasks with the upper-level subtasks when generating subtasks; An adversarial unit assignment module, used to assign the subtasks generated by the subtask generation module to each adversarial unit using deep reinforcement learning; The subtask execution module is used to execute the subtasks output by the adversarial unit allocation module, and update and push local environment observations to neighbors based on the execution results; The autonomous decision-making module is used to make local autonomous decisions when the confrontation unit is disconnected from the superior command center, that is, to determine the tasks that the local confrontation unit needs to perform.

2. A general decision-making method for time-varying networks, characterized in that: The steps include: 1) The command center uses the subtask generation module to take the environment state as input and generate subtasks under the constraints of the overall confrontation goal to maintain the consistency of the subtasks and the overall confrontation goal; 2) The command center uses deep reinforcement learning to assign subtasks to each adversary unit through the adversary unit assignment module; 3) Each adversarial unit queries the communication table through the decision architecture selection module before making a decision to determine whether the agent is disconnected from the command center: If there is no disconnection, agent i sends part of its own observations (ob i ), the command center receives some observation values ​​ob i Then, step 1) is executed periodically to make a centralized decision and assign subtasks to the agents corresponding to the received observations; Otherwise, the agent on the lower-level confrontation unit makes an autonomous decision through the autonomous decision-making module and executes step 4); 4) The adversarial unit determines whether there are any unfinished subtasks: If there are unfinished local subtasks, continue to execute the subtasks assigned by the command center through the confrontation unit allocation module; If the local subtask has been completed, but other subtasks in the original group have not been completed, other subtasks will be supported through the contract network protocol; If both the local and original group subtasks have been completed, a new subtask is autonomously generated through the subtask generation module; 5) The subtask execution module implements the subtasks determined in step 4), updates the local environment observations and pushes them to neighbors.

3. The method according to claim 2, characterized in that In step 1), the command center generates subtasks under the constraints of the overall confrontation goal through the subtask generation module. The implementation steps include the following: 1a) Reinforcement learning strategy network GN1 = π for subtask generation θ To train: 1a1) Set the type of units involved in the confrontation between the two sides at time t to type t 、Number count t 、location t The feature set s t ={(type t ,count t ,location t )} and the overall confrontation target G total are all input into the reinforcement learning strategy network, and the overall task action T is output action ,Task location T location ,Task Objective T target , and task time T time ; 1a2) Adopt a heuristic subtask generation strategy to generate the initial subtask set through expert knowledge in is the i-th initial subtask including attack and defense, 0≤i≤n, n is the total number of subtasks; 1a3) Initialize the reinforcement learning strategy network parameters θ; 1a4) Randomly generate subtasks T from the initial subtask set i The collection of local subtask space T local : Among them, δ is the preset local adjustment range, Δ i is a random mutation vector, distance(·) represents T i and The Euclidean distance between these two vectors; 1a5) According to the generated subtask set T = {T1, T2, ..., T i ,…,T n } and the current state s t Calculate the subtask set reward score Score(T,s t ): Among them, w i For task T i The weight of Task(T i ,s t ) is task T i score; 1a6) Update the reinforcement learning policy network parameters θ by gradient descent based on the reward score: Among them, α is the learning rate that determines the size of the parameter update step. represents the gradient of parameter θ, and E[·] represents the expectation; 1a7) Repeat steps 1a4) to 1a6) until the reinforcement learning policy network converges to obtain a trained reinforcement learning policy network; 1b) Use the trained reinforcement learning policy network for reasoning and generate subtasks under the overall adversarial goal constraint: 1b1) Set the type of units involved in the confrontation at time t1 to type t1 、Number count t1 、location t1 Composition feature set t1 : s t1 ={(type t1 ,count t1 ,location t1 )} 1b2) s t1 and the overall confrontation target G total Input into the trained reinforcement learning strategy network, and output the subtask set T at time t1 that meets the overall goal t1 : in, is the i-th subtask at time t1, including task actions Mission Location Mission Objectives and task time 4. The method according to claim 2, characterized in that: The command center in step 2) uses deep reinforcement learning to assign subtasks to each adversarial unit through the adversarial unit assignment module. The implementation steps include the following: 2a) Training the reinforcement learning policy network for adversarial unit allocation: 2a1) Initialize the existing reinforcement learning policy network N2=π using random initialization, Xavier initialization, and He initialization methods θ1 Parameters θ1 and evaluation network parameters φ; 2a2) Select the existing target network Q′ φ and Q Network φ , and the target network Q′ φ The parameters of the Q network are initialized to φ The parameters are the same as: Q′ φ ←Q φ 2a3) Set the environmental state s t , subtask set {T1,T2,…,T n } and the set of available adversarial units {U1,U2,…,U m } Input reinforcement learning policy network π θ1 , output action a t : a t ={group j ((T j1 ,U ja ),(T j2 ,U jb ),…,9T jk ,U jx ),…,(T jn ,U jy ))} Among them, group j (·) represents the combination of the jth group of cooperative confrontation units, (T jk ,U jx ) means that in the jth group, the number U jx The adversarial unit is assigned to T jk subtasks; 2a4) Update each assigned subtask T according to the assignment result i The status is completed, and the environment status s is updated according to the task actions, task locations, task goals, and task times of these newly completed subtasks. t+1 ; 2a5) Calculate the target Q value y of the reinforcement learning policy network t : y t =r t +γQ φ (s t+1 ,π θ1 (s t+1 )) Among them, the target Q value y t is the future discounted reward that the agent expects to receive after taking an action; r t is the immediate reward obtained by the agent from the environment at time t; γ is a discount factor used to weigh the importance of immediate rewards and future rewards; Q φ (s t+1 ,π θ1 (s t+1 )) is in state s t+1 Next, according to the current reinforcement learning strategy network π θ1 The Q value corresponding to the selected action; 2a6) Calculate the predicted Q value of the evaluation network Among them, Q φ (s t ,a t ) is the evaluation network with parameter φ in state s t and action a t The Q value output under 2a7) Calculate the loss function L used to update the Q network through the back-propagation algorithm Q (φ): 2a8) Minimize the loss function through the gradient descent algorithm and update the parameter φ of the evaluation network so that its predicted Q value is closer to the target Q value: Where α is the learning rate that determines the step size of parameter update, is the loss function L Q (φ) Gradient with respect to parameter φ; 2a9) Compute reinforcement learning policy network π θ1 The loss function L π (θ1): L π (θ)=-Q φ (s t ,p θ1 (s t )) Among them, Q φ (s t ,π θ1 (s t )) is state s t According to the reinforcement learning strategy network π θ1 The Q value corresponding to the selected action; 2a10) Calculate the loss function L π (θ1) Gradient of the reinforcement learning policy network parameter θ1 And update the parameter θ1: 2a11) Update the target network parameter φ′ by the following formula: φ′←τφ+(1-τ)φ′0<τ<1 Among them, τ is the soft update coefficient; 2a12) Repeat steps 2a4) to 2a11) until the reinforcement learning policy network π θ1 Converge and obtain a trained reinforcement learning strategy network; 2b) The environmental state s at time t2 t2 , subtask collection and the set of available adversarial units {U1,U2,…,U m } Input the trained reinforcement learning strategy network π θ , output action a t2 : in, Indicates that at time t2, the number U jx The opposing unit is assigned to subtasks.

5. The method according to claim 2, characterized in that: Step 3) Each adversarial unit queries the communication table through the decision architecture selection module before making a decision to determine whether the intelligent agent is disconnected from the command center. The implementation steps include the following: 3a) Each agent i observes the environment according to period T0 and obtains the local observation value O i ; 3b) Each agent i that performs a task creates a communication table to record its direct network connections with other neighboring agents j. Direct connection to the command center and hop count TTL i ; If agent i is directly connected to agent j, then otherwise Similarly, if agent i is directly connected to the command center, then otherwise 3c) All agents follow the cycle T com Exchange the communication table information with all neighbors in the communication table to obtain the direct adjacency matrix M of the entire system: 3d) Each agent i queries the matrix M and performs a depth-first search: For each neighbor agent j, if Then, we further recursively search all reachable neighbors k of agent j according to matrix M, and set Repeat this recursive process until all directly connected agents and their neighbors have been traversed; examine If the value of It means that agent i can communicate with the command center directly or indirectly; otherwise It means that agent i is disconnected from the command center.

6. The method according to claim 2, characterized in that In step 4), other subtasks are supported through the contract network protocol, and the implementation steps include the following: 4a) The opposing unit with the unfinished subtask broadcasts the task requirements, including capabilities and location information, and the opposing unit decides whether to accept based on its own capabilities; 4b) Select the corresponding opposing units that received the message first among the opposing units that accepted the task, and send them a task confirmation message; 4c) Add the selected opposing unit to its own sub-environment to perform the task.

7. The method according to claim 2, characterized in that In step 4), a new subtask is autonomously generated by the subtask generation module, and the implementation steps include the following: 4d) On the adversarial unit, the existing reinforcement learning strategy network GN3 = π is used for autonomous feeling θ3 To train: 4d1) Set the type of opposing units involved in the confrontation at time t to type t 、Number count t 、location t The feature set s t ={(type t ,count t ,location t )} and the overall confrontation target G total are all input into the reinforcement learning strategy network, and the overall task action T is output action ,Task location T location ,Task Objective T target , and task time T time ; 4d2) Adopt a heuristic subtask generation strategy to generate an initial subtask set through expert knowledge in is the i-th initial subtask including attack and defense, 0≤i≤n, n is the total number of subtasks; 4d3) Initialize the reinforcement learning strategy network parameters θ3; 4d4) Randomly generate subtasks T from the initial subtask set i The collection of local subtask space T local : Among them, δ is the preset local adjustment range, Δ i is a random mutation vector, distance(·) represents T i and The Euclidean distance between these two vectors; 4d5) According to the generated subtask set T = {T1, T2, ..., T i ,…,T n } and the current state s t Calculate the reward value score of the subtask set generated by autonomous decision-making Score1(T,s t ): Among them, w i For task T i The weight of Task(T i ,s t ) is task T i Score, AT i To autonomously generate subtask T i The corresponding action vector; The action vector corresponding to the subtask assigned by the superior to this adversarial unit when the connection is not disconnected before the subtask is generated autonomously. It is the cosine similarity between the action vectors corresponding to the subtask generated by the self-decision after disconnection from the superior and the subtask assigned by the nearest superior. The calculation formula is: Where A and B are the two vectors to be compared, A·B represents the dot product of the vectors, also known as the inner product, which is the sum of the products of the corresponding elements of the two vectors; ‖A‖ and ‖B‖ represent the modulus of the vectors, which can be obtained by calculating the sum of the squares of each vector element and then taking the square root; 4d6) Update the reinforcement learning policy network parameters θ3 by gradient descent based on the reward value score: Where α is the learning rate used to determine the step size of parameter updates. represents the gradient of parameter θ3, and E[·] represents the expectation; 4d7) Repeat steps 4d4) to 4d6) until the reinforcement learning strategy network converges to obtain a trained reinforcement learning strategy network N2 for autonomous decision-making; 4e) Perform the reasoning phase on the adversarial unit, and output a set of subtasks that meet the current overall goal based on the observation of the other party, which includes the following steps: 4e1) Set the type of units involved in the confrontation at time t3 to type t3 、Number count t3 、location t3 Composition feature set t3 : s t3 ={(type t3 ,count t3 ,location t3 )} 4e2) s t3 and the overall confrontation target G total Input to the trained reinforcement learning strategy network GN3, and output the subtask set T at time t3 that meets the overall goal t3 : in, is the i-th subtask at time t3, including task actions Mission Location Mission Objectives and task time 8. The method according to claim 2, characterized in that: In step 5), the subtask execution module implements the subtask determined in step 4), and updates and pushes the local observation value. The implementation steps include the following: 5a) Each assigned task adversarial unit i performs an action and updates the environment observation o i ; 5b) The decision-making and execution-completed agent i pushes its local observation value o to the neighboring nodes in the communication table i and the current time.

Images