Hierarchical reinforcement learning method for uncertainty auxiliary task in adversarial scenario

Abstract

The application proposes a hierarchical reinforcement learning method for uncertainty auxiliary tasks in an adversarial scenario. First, the task allocation stage of the upper layer is performed, the agent first acquires global environment information, then extracts important auxiliary task information, and then extracts important main task information based on the extraction, and finally learns the task allocation strategy combined with the information of other agents; Then, the task execution stage of the lower layer is performed, the agent first constructs a dedicated sub-environment according to the allocation result, then learns the task execution sequence in the sub-environment, and finally executes specific actions. The hierarchical reinforcement learning algorithm proposed in the patent can more efficiently help the multi-agent system learn how to execute uncertain auxiliary tasks, by first learning the multi-agent uncertainty auxiliary task allocation strategy of the upper layer, and then learning the single-agent uncertainty auxiliary task execution strategy of the lower layer, respectively solving the influence of uncertain auxiliary tasks on the group and the individual, which can effectively reduce the complexity of the problem.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of autonomous unmanned systems, and particularly relates to a hierarchical reinforcement learning method for uncertain auxiliary tasks in confrontation scenarios. BACKGROUND

[0002] In recent years, different kinds of autonomous unmanned systems, such as unmanned aerial vehicles, unmanned vehicles, bionic robots, etc., have been gradually applied to various complex and uncertain environments such as military, anti-terrorism, business, disaster relief, etc. to perform tasks. In these tasks, in addition to the "main task" directly related to the optimization target, there is also an "auxiliary task" indirectly related to the optimization target. The execution of the auxiliary task can improve the execution efficiency of other tasks within its influence range to a certain extent (for example, in an electronic confrontation scenario, attacking enemy ships is the main task, and shielding enemy electromagnetic interference is the auxiliary task). Therefore, according to the correlation between the main task and the auxiliary task, each agent in the multi-agent system needs to be assigned a corresponding task, and a corresponding task execution strategy needs to be designed for it to improve the utility of task execution.

[0003] In actual confrontation scenarios, the auxiliary ability of the auxiliary task is uncertain, mainly reflected in the uncertainty of the gain of the execution efficiency of other tasks within its influence range. In this case, due to the lack of modeling of such uncertain auxiliary tasks in previous algorithms, the system may fall into a local optimal solution. When designing a targeted algorithm, there are two challenges: first, when designing a task allocation strategy for a multi-agent system, the uncertain auxiliary task and its impact on the main task need to be considered, which increases the dimension of the problem solution; second, when considering the task execution strategy, the task execution order of the agent needs to be adjusted adaptively according to the uncertainty of the auxiliary task, making the solution space of the problem complex.

[0004] To solve this problem, the present application proposes a hierarchical multi-agent reinforcement learning method to solve the task allocation and task execution problems caused by the influence of uncertain auxiliary tasks on the group and the individual, respectively, in order to reduce the complexity of the problem and the size of the solution space. First, the upper reinforcement learning is used to solve the task allocation problem caused by the influence of uncertain auxiliary tasks on the entire multi-agent system, as shown in Figure 1 Each agent is assigned a part of the task, and some auxiliary tasks are discarded due to weak auxiliary ability. The network structure of this layer of reinforcement learning is as shown in Figure 2As shown, first, each agent embeds environment information into itself and task information, then extracts important information from all auxiliary tasks, and based on this, extracts important information from all main tasks, after considering the information of other agents, the agent will decide which tasks to assign to itself, after sampling the highest revenue allocation method, back propagation is used to update network parameters, and iteration is continuously performed to maximize revenue. After each agent gets the tasks assigned to it, they need to build a dedicated sub-environment according to these tasks, and then the agent can only observe and act within its dedicated sub-environment, which can reduce the observation space of the lower reinforcement learning. Lower reinforcement learning is used to solve the problem of task execution caused by the influence of uncertain auxiliary tasks on a single agent, such as Figure 3 As shown, the agent autonomously decides the execution order of the tasks within its sub-environment; as Figure 4 As shown, the agent needs to observe the risk level of its surroundings and try to move to a low-risk area.

[0005] The task allocation and task execution algorithm of the multi-agent system in the adversarial scenario is to make the system have a small agent loss rate and a low task completion time. When there are uncertain auxiliary tasks in the environment, the problem becomes much more complex. For the entire multi-agent system, the auxiliary task with strong auxiliary ability needs to be assigned to the appropriate agent; for each agent, the auxiliary task with strong auxiliary ability needs to be executed first, and then the other tasks within its influence range are executed. Therefore, we hope to use hierarchical reinforcement learning to divide the above complex problem into two sub-problems of task allocation and task execution, to better handle the uncertainty of auxiliary tasks, and to reduce the complexity of the problem and the size of the solution space. SUMMARY

[0006] Technical problem: The purpose of the present application is to propose a hierarchical reinforcement learning method for uncertain auxiliary tasks in an adversarial scenario. The method is divided into two layers. The upper layer reinforcement learning is used to solve the task allocation problem caused by uncertain auxiliary tasks. In this layer, the system may discard some auxiliary tasks and assign the remaining tasks to each agent. The lower layer reinforcement learning is used to solve the task execution problem caused by uncertain auxiliary tasks. In this layer, the system will decide the specific order of task execution. This method solves the problem that direct task execution will cause the solution space to be too complex when there are uncertain auxiliary tasks in the adversarial environment.

[0007] Technical solution: In the adversarial environment, multi-agent system needs to perform some tasks, among which there are auxiliary tasks with uncertain auxiliary ability, that is, the gain of the auxiliary task to the execution efficiency of other tasks within its influence range is uncertain. In order to adapt to this uncertainty and maximize the utility of auxiliary tasks, the agent needs to first decide which auxiliary tasks to assign to which agent, and then decide the order of executing these tasks. At the same time, in order to reduce the complexity of the problem and the size of the solution space, we use hierarchical reinforcement learning to decompose the problem into task allocation and task execution two sub problems, the main technical scheme of this hierarchical reinforcement learning method is as follows:

[0008] First, the upper layer reinforcement learning, this layer is used to solve the task allocation problem caused by the influence of the uncertainty of auxiliary tasks on the whole multi-agent system. For each agent, the input of this layer is all the information of itself, all the information of all tasks, global environment information, and the information of other agents, and the output is the task allocated to each agent respectively, each agent will be allocated to a part of all tasks. Each agent in the multi-agent system first embeds the environment information into itself and task information, then extracts the important auxiliary task information from all auxiliary tasks, where the auxiliary task close to itself may be important, and then based on this, extracts important main task information, where the main task close to itself and important auxiliary task may be important, and the task in the high-risk position may be unimportant, then the information of other agents is fused in, so that the information of itself and important tasks to itself and the information of other agents can be represented as a high-dimensional vector, then each agent decides which tasks to execute from this high-dimensional vector. Each agent has a Q function, and the Q functions of all agents are monotonically added to constitute the Q function of the whole agent cluster, that is, when the Q function of each agent increases, the overall Q function also increases, and when the Q function of each agent is optimal, the Q function of the whole agent cluster is also optimal. The reward function of each agent is related to the task completion time and the survival rate of the agent, specifically, the shorter the expected time for each agent to complete the assigned task, the higher the expected survival rate of the agent, the greater the reward.

[0009] Then is the lower layer reinforcement learning, which is used to solve the task execution problem caused by the influence of the auxiliary task uncertainty on the single agent. This layer first constructs the sub-environment. The last layer reinforcement learning outputs the task set to which each agent is assigned, and the agent constructs its own exclusive sub-environment according to the set, which only includes itself and the assigned tasks, and does not include other agents and unassigned tasks. The sub-environments of different agents will have overlapping parts, because there may be high-risk areas in the environment, and agents may need to take a detour to walk through a safe area to execute tasks. Therefore, the sub-environment construction needs to be as large as possible under the condition of not including other entities, in order to provide space for possible detours, where agents and tasks are collectively referred to as entities. Through sub-environment construction, the observation space of the lower layer reinforcement learning can be greatly shortened, and the learning efficiency can be improved. The input of this layer for each agent is all the information of itself, all the information of the assigned tasks, and the risk index of each position in the sub-environment. The output is the speed vector v of each agent a . This layer of reinforcement learning is used to let each agent autonomously decide the specific task execution path. Since the sub-environment construction in the last step has been completed, each agent now has no influence on each other in their respective sub-environments, and there is no intersection of entities in each sub-environment, so the reinforcement learning of this layer is transformed into a single-agent reinforcement learning problem, and the difficulty of the problem is greatly reduced. The reward function design of this layer is similar to that of the upper layer reinforcement learning, the difference is that the upper layer reinforcement learning network is updated every N t steps, and this layer is updated every 1 step.

[0010] Advantages:

[0011] (1) Improve the adaptability of the multi-agent system to auxiliary task uncertainty: In the upper layer reinforcement learning, each agent first extracts important auxiliary task information that is important to itself, and based on this, extracts important main task information. This can discard auxiliary tasks with a cost greater than the benefit in the assignment stage, and can also represent auxiliary tasks and other tasks within their auxiliary range together. In the lower layer reinforcement learning, the agent will observe uncertain auxiliary tasks in real time during task execution, and try to choose to execute when its auxiliary ability is large. This can maximize the utility of auxiliary tasks in the execution stage.

[0012] (2) Reduce the total time of multi-agent system to complete the task: first, the upper reinforcement learning will assign each agent a corresponding task, and consider the expected time of each agent to complete the task during the assignment, to ensure that their expected task completion time is close, so as to make the first optimization of the task completion time of the multi-agent system. In the lower reinforcement learning, the agent will master the optimal route to complete the task in the sub-environment through continuous training, so as to shorten the specific time of completing the task as much as possible, which is the second optimization. Through the two optimizations, the total time of the multi-agent system to complete the task can be effectively reduced.

[0013] (3) Reduce the loss rate of multi-agent system: first, the upper reinforcement learning will assign each agent a task set, and consider the risk that each agent may encounter when performing the task afterwards during the assignment, that is, to ensure that the expected risk they encounter when performing the task is close, so as to make the first optimization of the loss rate of the multi-agent system. In the lower reinforcement learning, the agent will master the optimal route to complete the task in the sub-environment through continuous training, and try to avoid high-risk areas to reduce its loss rate, which is the second optimization. Through the two optimizations, the loss rate of the multi-agent system can be effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The task assignment result calculated by the upper reinforcement learning for the whole agent cluster is shown.

[0015] Figure 2 The network structure of the upper reinforcement learning is shown.

[0016] Figure 3 The order of the agent performing the task in a certain sub-environment is shown.

[0017] Figure 4 The specific moving path of a certain agent performing a certain task is shown

[0018] Figure 5 It is the main principle diagram of the method of the application.

[0019] In Figure 1 , the circle represents the agent, the square represents the main task, and the triangle represents the auxiliary task. The agent will tend to assign the task near it to itself, and abandon the auxiliary task with a cost greater than the auxiliary effect. In Figure 2 , the square solid black block represents the dangerous environment, and the circle represents the other agent. The agent will consider the global environment information to perform task assignment in turn. In Figure 3 , the dashed circle represents the auxiliary range of the auxiliary task, and the arrow represents the order of the agent performing the task. The agent will prefer to perform the auxiliary task with strong auxiliary ability close to itself, and then perform the main task around it.Figure 4 In the figure, the arrow represents the path of the agent, and the agent will bypass the high-risk area during the process of going to a certain task. DETAILED DESCRIPTION

[0020] The present application will be further illustrated in conjunction with the accompanying drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0021] As shown in the figure, the present application provides a hierarchical reinforcement learning method for uncertainty auxiliary tasks in an adversarial scenario.

[0022] The overall process of the method is as follows: each agent a first embeds the environmental risk degree information into the entity information, then extracts the task information important to itself from all tasks, and after considering the information of other agents, autonomously decides the task set assigned to itself Record this as a task assignment b a The task assignment of all agents is combined into the task assignment strategy set of the entire agent cluster A Then, each agent calculates its exclusive sub-environment z a according to its b a Finally, each agent autonomously decides the movement path required to complete the task in its own sub-environment. Next, the two stages of the method are described respectively.

[0023] (1) Task assignment stage: First, the environmental information is acquired. The environment is a two-dimensional continuous space, each location has a coordinate (x, y), and each location has a risk index r, satisfying 0≤r≤1. The greater the risk index, the lower the survival rate of the agent. The risk index is dynamic, and at the initialization of the environment, the risk index of each location is randomly sampled within the range of 0 to 1. After that, every time step, the risk index r of each location is added by a risk increment Δr, Δr ~ N(0, 1), if r+Δr>1, then r=1, if r+Δr<0, then r=0. The more dangerous the location of the agent, the longer it stays in the environment, the lower the survival rate of the agent. The survival rate of the agent is represented as: where represents the risk index of the location of agent a at time t, which means that from the initial time, every time step, the agent calculates the safety of the current location Then, the safety from t=0 to the current time is integrated as the survival rate of agent a at the current time.

[0024] The agent cluster A is composed of many agents a∈A, each agent has the following attributes: position p pos , velocity p vel , and capability size aabl and survival ratea sur The task set T consists of many tasks t∈T, each of which has the following properties: type t type , position t pos , size t size If the task is an auxiliary task, it also has the following properties: auxiliary range t range , auxiliary coefficient t co ,When the auxiliary task is completed, its auxiliary ability is reflected. The auxiliary range is expressed as the auxiliary task's ability to take itself as the center of the circle, t range The impact on other tasks within the circle of radius is to reduce the size of all tasks except itself, including auxiliary tasks, so that t size =t size *t co The more dangerous the auxiliary task is, the smaller its auxiliary range is. Therefore, when the risk of the auxiliary task is r = 0, we record its auxiliary range as When the environment changes, there are

[0025] This multi-agent reinforcement learning process can be modeled as a Markov game (S, U, P, r, γ). S represents the state space of reinforcement learning, and each agent a has a state s. a , the states of all agents constitute the state of the entire multi-agent system, that is, S = ∪s a , s a It is the agent a based on its own observation O a Inferred, O a The information includes its current location p pos , its own ability size a abl and survival ratea sur , also includes the information of all tasks T, that is, the type t of each task t∈T type , position t pos , size t size , the risk index R of the location t If a task is an auxiliary task, its auxiliary range t will also be observed range , auxiliary coefficient t co , and finally includes the position of other agents pos and the survival rate of other agents sur U represents the action space of reinforcement learning. The action of the agent in this layer is the allocation strategy U=B, that is, each agent a makes its own decision, and eventually the entire agent set A will decide on a set of task allocation solutions B, in which each agent has an independent allocation strategy. satisfy where n represents the number of agents, represents the mth task assigned to agent a, where m < |T|. P represents the state transition distribution, denoted as probability P(S'|S, U). r is the reward function, each agent has an independent reward function, and their sum constitutes the reward function of the multi-agent system. γ represents the discount factor, which measures the importance of future rewards, and its value ranges between 0 and 1, i.e. the future reward is multiplied by the exponential decay function of γ, so that the value of future rewards decays exponentially over time, in order to maximize the overall reward.

[0026] The general process of this stage is as follows: first, embed the environment information into the entity information, where the agent and the task are collectively referred to as the entity, to obtain the environment information near the agent and the task; then use the FC+MHA network to extract important auxiliary task information from all auxiliary tasks, where the auxiliary task that is closer to itself, has stronger auxiliary ability, and is located in a lower risk position may be important to itself; then use the FC+MHA network based on this result to extract important main task information from all main tasks, where the main task that is closer to itself and is located in a lower risk position may be important, and if a main task has a high correlation with the important auxiliary task information just extracted, it may also be important, where if a main task is within the auxiliary range of an important auxiliary task, it may have a high correlation with the auxiliary task information; then consider the information of other agents, so as to obtain a high-dimensional vector containing its own information, important task information and other agent information; finally, learn from this high-dimensional vector which tasks to assign to itself. Next, the process is described in detail.

[0027] First, each agent will obtain all entity information and global environment risk degree information, and then embed the environment risk degree information into the entity information. The purpose of this step is that not every piece of environment risk degree information in the environment is worth paying attention to, and we tend to pay attention to the environment information near the entity, i.e. the risk degree information on the possible movement path of the agent during the execution of the task. For each entity, the environment information near it refers to the environment risk degree information within a certain length radius circle with itself as the center.

[0028] Then each agent extracts the important auxiliary task information from all auxiliary tasks. If an auxiliary task is close to the agent, has strong auxiliary ability, and is not in a high-risk position, the auxiliary task may be important to the agent. At the same time, auxiliary tasks can assist auxiliary tasks, that is, they can also reduce the task size of auxiliary tasks. When an auxiliary task is assisted, the completion cost is reduced because the task size is reduced, so the agent may obtain higher benefits by executing it again. Therefore, the auxiliary tasks near the important auxiliary tasks to the agent may also be important to the agent. Then all auxiliary task information is represented as a high-dimensional vector, in which the proportion of important auxiliary task information to the agent in the vector is larger, and the proportion of unimportant auxiliary task information to the agent in the vector is smaller. This step uses the multi-head attention (MHA) mechanism to analyze the relevance between the current agent and the auxiliary tasks and the relevance between the auxiliary tasks, so as to judge which auxiliary tasks are important to the current agent.

[0029] The input matrix of this step is the auxiliary task vector set T aux ∈R n×d , and the output matrix is the grouped auxiliary task group T' aux ∈R d′ , where n is the length of the input, that is, the number of auxiliary tasks, and d and d' are the dimensions of the input and output, respectively. We use h attention heads, and the output dimension of each head is d h =d' / h. Then the calculation process of MHA is as follows:

[0030] Linearly transform the input matrix T aux to obtain the query matrix key matrix and value matrix The formula for linear transformation is: Q=T aux W Q , K=T aux W K , and V=T aux W V , where are trainable parameter matrices. Then, the dot product attention calculation is performed on the query matrix Q and the key matrix K to obtain the attention matrix A∈R h×n×n . Specifically, for each attention head i∈[1,h], we calculate the attention weight matrix A i ∈R n×n , and the formula is:

[0031]

[0032] where WQi ∈R n×n and W Ki ∈R n×n are trainable parameter matrices.

[0033] Then the attention matrix A and the value matrix V are multiplied to obtain the output matrix of each head Specifically, for each head i∈[1, h], we calculate the output matrix of head i The formula is: head i = A i VVV i , where are trainable parameter matrices. Then the output matrix of each head head i is concatenated along the output dimension to obtain the matrix

[0034] head∈R n×d′ The formula is: head = [head1; head2; …; head h ]W O , where [head1; head2; …; head h ] represents concatenating the output matrix of each head along the output dimension, are trainable parameter matrices.

[0035] Finally, the matrix head is subjected to residual connection and normalization to obtain the final output matrix T′ aux ∈R d′ The formula is: T′ aux = LayerNorm(T aux + head), where LayerNorm represents layer normalization on the input, and T aux + head represents the operation of residual connection.

[0036] After the agent extracts the important auxiliary task information of itself, the important main task information of itself needs to be extracted, wherein the main task that is close to itself and has low risk of location may be important, and the main task that has high relevance with the auxiliary task information extracted in the previous step may also be important, wherein if the main task is within the auxiliary range of the important auxiliary task, it may have high relevance. This step also uses the MHA structure, which can obtain the relevance of the main task and the current agent, and also obtain the relevance of the main task and the important auxiliary task information, so as to judge which main task is important to the agent.

[0037] To date, we have expressed the current agent information, environmental information, important auxiliary task information and important main task information into a high-dimensional vector, and finally fused other agent information into the current vector to obtain all the information that the agent needs to consider in the task allocation stage, and expressed these information into a final high-dimensional vector. The agent needs to make a decision on which tasks to perform through this vector, i.e. complete the task allocation of itself. In this step, each agent continuously trains and backpropagates to enable the multi-agent system to find the optimal allocation strategy set B in the current state, i.e. find argmax B Q tot (S,B), and since each agent makes independent decisions, and satisfies for any a∈A, Q a The growth of Q tot , therefore, to find the overall optimal allocation strategy, it can be considered that when each agent a finds its own optimal allocation strategy b, i.e. finds argmax b Q(s,b), the overall allocation strategy set B is also optimal, which is the IGM (Individual-Global-Max) principle.

[0038] A memory bank is set in the training to store previously learned data, i.e. S t ,U t ,r t ,S t+1 Each time of training will randomly extract a part of data from the memory bank for learning, and through this way, the correlation between data can be broken, the data is more random, and the result is more accurate. θ represents the current network, represents the target network, θ is updated in real time, and periodically obtains the parameters of θ and updates itself, and this way of freezing network update is also a way to cut off the correlation.

[0039] Each agent a' needs to consider the allocation b a′ that has been made by other agents a before obtaining the task allocation b a , so as to perform iterative task allocation between agents; each agent needs to continuously optimize its Q function during training, and the Q function is expressed as:

[0040]

[0041] And the way to optimize the Q function is to continuously reduce the loss function during training, and the loss function is expressed as:

[0042]

[0043]

[0044] The principle of the reward function design is that the longer the expectation time for each agent to complete the task assigned to it, the higher the expectation probability of being damaged, and the less the reward it gets, and the more tasks it completes, the more the reward it gets, based on which the reward can be expressed as:

[0045]

[0046]

[0047] wherein represents the time for completing the task assigned to the agent a, represents the time for moving to the task, represents the time for executing the task, represents the risk degree of the location where the task assigned to the agent a is located, represents the damage rate of the agent in the process of executing the task, represents the size of the task assigned to the agent a.

[0048] For the case of a single agent executing a single task, the time for the agent a to complete the task t is divided into two parts, represents the time for the agent a to move to the task point t, d(a, t) represents the Euclidean distance between the agent a and the task point t, represents the time for the agent a to execute the task t,

[0049] After each agent learns its own assignment strategy, the agent cluster also needs to be evaluated as a whole to determine whether the current assignment strategy of all agents is optimal for the entire agent cluster, at which time, the Q function of the agent cluster needs to be optimized, which is expressed as:

[0050] Q tot (S, U) = r(S, U) + γE[maxQ tot (S', U')]

[0051] Similarly, optimizing this Q function is to continuously reduce its loss function in the training process, and the loss function is expressed as:

[0052]

[0053]

[0054] That is, each agent can find the optimal task allocation strategy bafor itself through continuous training, and the b a of all agents form the task allocation strategy set B of the entire agent cluster.

[0055] Since the reinforcement learning of the current layer is the upper layer, its output will affect the reinforcement learning of the lower layer, so each step of the current layer is equivalent to N t time steps of the lower layer, that is, the agent performs a distribution at the current layer, and the lower layer spends N t steps to execute the task of this distribution.

[0056] Every round, that is, every N t time steps, the current stage is repeated, that is, each agent represents the important information it needs as a high-dimensional vector, and then the multi-agent system makes a decision U to give the allocation strategy B, and after getting the corresponding reward r, it updates the parameters of the network θ through back propagation to optimize the next decision, so that it is constantly looped, and finally the multi-agent system can learn the optimal allocation strategy and make adaptive autonomous decisions for uncertain auxiliary tasks in a hostile environment.

[0057] (2) Task execution phase: In the previous phase, each agent has obtained a distribution result, that is, the task to which it is allocated, and then needs to generate a specific execution plan according to the allocated task, that is, the order and movement path of the agent to execute the task. Since there are risk factors R in the environment, the agent needs to choose to move in a safe area as much as possible to ensure its safety, so it needs to observe the risk indicators in the environment. The most ideal case is that each agent completely observes the risk indicators of the entire environment and chooses a route with the lowest risk, even if this route may be a long detour. However, in fact, the agent only needs to observe the environment around itself and its allocated task, which not only greatly reduces the size of the observation space and speeds up the learning speed, but also avoids the situation of a long detour.

[0058] Based on the above considerations, for each agent a, define E a = a U b a , which represents the entity set of agent a and its allocated task b a , and then use the alpha shapes algorithm to calculate the contour points of E a , denoted as L a ; To expand the sub-environment, we find for each element l a in L a , the other point e' in the environment that is closest to it but does not belong to E a ​a , and calculate l a The minimum distance from the map border, if this minimum distance is less than l a to e′ a The distance is, then the point on the map boundary corresponding to this minimum distance is added to the set E a ', otherwise e' a Add to E a ′, the advantage of doing this is that if l a Near the map boundary, the map boundary point is used as the boundary point of the sub-environment; then the convex hull method is used to calculate E a ′, and record this contour point set as L a ′, and finally L a By connecting the points in ′ in sequence, we get the boundary of the sub-environment, and the range surrounded by this boundary is the sub-environment z that the agent a can actually observe. a .

[0059] For each agent, it can and can only observe all the information in its sub-environment, and cannot observe the sub-environment of other agents. In this way, the entire environment can be divided into many sub-environments, which will overlap with each other and satisfy Where n represents the number of agents and map represents the entire environment.

[0060] This single-agent reinforcement learning process can be described as a Markov decision process (S, U, P, r, γ). Since each agent has its own dedicated sub-environment, each agent does not affect each other. In the sub-environment, the problem is simplified to a single-agent reinforcement learning problem. The state S of the agent is inferred based on its own observation O, which includes the agent's current position p. pos , current speed p vel , capacity size a abl , survival rate a sur , the risk index R of the sub-environment a , the set of tasks assigned to b a The type of each task t in type , position t pos and size t size If a task is an auxiliary task, its auxiliary scope t is also included range , auxiliary coefficient t c o, the action space U is the velocity vector v of each agent aP represents the state transition distribution, denoted as P(S'|S, U). Here, the reward function r of each agent does not affect each other. γ represents the discount factor, which measures the importance of future rewards, and its value ranges between 0 and 1.

[0061] Since every N t steps, the previous layer of reinforcement learning generates a new allocation strategy, so the task to which each agent is allocated may change, the number of tasks to which each agent is allocated may change, and the observed sub-environment z a may also change, that is, the size of the observation space O of each agent may dynamically change. In order to handle this situation, an attention mechanism needs to be integrated. When the length of the input vector set of reinforcement learning changes, the attention mechanism can convert it into a fixed-length fusion feature vector.

[0062] The agent a outputs a specific action u a during the execution of the task, that is, v a , the reward function r b only depends on the sub-environment information, and is defined as: where is the global state set obtained by the agent cluster after executing the optimal task execution strategy under the given task allocation strategy B.

[0063] After each agent obtains its specific sub-environment z a , since the sub-environment of each agent is different, the size of the observation space of each agent is different, and for a certain agent, its sub-environment will change over time, that is, the size of its observation space will dynamically change, therefore, an attention mechanism needs to be used to handle this variable observation space; for each agent, it needs to continuously optimize its network parameters θ during training, that is:

[0064]

[0065]

[0066] In the formula, τ represents the state-action sequence generated by the interaction between the agent and the sub-environment during training, π θ (τ) represents the probability of sequence τ under the current network parameter θ, and τ can be represented as:

[0067] τ = s1, u1, s2, u2, …, s t , u t

[0068] The policy gradient of this layer of reinforcement learning is represented as:

[0069]

[0070] And use this gradient to update the network parameters θ, that is:

[0071]

[0072] In the sub-environment, the agent receives a negative reward, that is, a punishment, every time step, which is designed to make the agent learn to complete all tasks as soon as possible in the process of being constantly punished, so as to not only reduce the task completion time, but also improve the survival rate of the agent, the reward function is designed as:

[0073]

[0074] Where represents the Euclidean distance between the agent a and a certain task assigned to it R a represents the risk degree of the location where the agent a is located.

[0075] Every time step, the layer reinforcement learning reacquires the sub-environment information to obtain a new observation O, then makes a decision U, and after obtaining the corresponding reward r, it updates the network θ parameters through the loss function for back propagation to optimize the next decision, so as to continuously cycle, finally, the agent can learn the optimal task execution path, and can make adaptive autonomous decisions for uncertain auxiliary tasks in the adversarial environment.

Claims

1. A hierarchical reinforcement learning method for an auxiliary task under uncertainty in adversarial scenarios, characterized in that: The method comprises, firstly, a task allocation phase of upper-layer reinforcement learning, in which an agent first acquires global environment information, then extracts important auxiliary task information for itself using an FC+MHA network, extracts important main task information, and finally learns a task allocation strategy in combination with information of other agents; then, a task execution phase of lower-layer reinforcement learning, in which the agent first constructs a dedicated sub-environment according to the allocation result, then learns a task execution sequence in the sub-environment, and finally executes specific actions; wherein the upper-layer reinforcement learning is used to solve a task allocation problem caused by an influence of uncertain auxiliary tasks on the whole multi-agent system, and the lower-layer reinforcement learning is used to solve a task execution problem caused by an influence of uncertain auxiliary tasks on a single agent; In the task allocation phase, the environment information is obtained. The environment is a two-dimensional continuous space. Each position has a coordinate (x, y). Each position has a risk index r, which satisfies 0≤r≤1. The greater the risk index, the lower the survival rate of the agent. The risk index is dynamically changing. When the environment is initialized, the risk index of each position is randomly sampled within the range of 0 to 1. Then, every time step, the risk index r of each position is added by a risk increment Δr ~ N(0, 1). If r+Δr>1, then r=1. If r+Δr<0, then r=0. The agent cluster A is composed of multiple agents a∈A. Each agent has the following attributes: position p pos , velocity p vel , ability a abl , survival rate a sur . The task set T is composed of multiple tasks t∈T. Each task has the following attributes: type t type , position t pos , size t size . If the task is an auxiliary task, it also has the following attributes: auxiliary range t range , auxiliary coefficient t co . The auxiliary range is represented as a circle with the auxiliary task as the center and t range as the radius. The influence is that it can reduce the size of all other tasks within the range except itself, so t size =t size *t co . The smaller the auxiliary range of the auxiliary task, the more dangerous the position of the auxiliary task. Therefore, when the risk r=0, the auxiliary range is recorded as When the environment changes, time for agent a to complete task t is split into two parts, denotes the time for agent a to move to task point t, d(a, t) denotes the Euclidean distance between agent a and task point t, denotes the time for agent a to perform task t, The survival rate of the agent is represented as: wherein represents the risk index of the position of the agent a at time t. Each agent a first embeds the environmental risk information into the entity information, then extracts the task information that is important to itself from all tasks, and autonomously decides the task set assigned to itself after considering the information of other agents. Recorded as a task assignment b a , the task allocation of all agents is combined into the task allocation strategy set of the entire agent cluster A Afterwards, each agent is assigned a a Calculate its exclusive sub-environment z a ,Finally, each agent autonomously decides the movement path required to complete the ,task in its respective sub-environment.

2. The hierarchical reinforcement learning method for auxiliary tasks under uncertainty in adversarial scenarios according to claim 1, wherein: Each agent a' considers the assignments b made by other agents a before obtaining its assignment b a′ Each agent a' considers the assignments b made by other agents a before obtaining its assignment b a The iterative assignment of tasks between agents is performed, each agent continually optimising its Q function during training, which is represented as: The way to optimize the Q function is to continuously reduce the loss function during the training process, and the loss function is represented as: After each agent learns its own allocation strategy, the agent cluster also needs to be evaluated as a whole to determine whether all the allocation strategies are optimal for the whole agent cluster, that is, to optimize the Q function of the agent cluster, which is represented as: Optimizing this Q function is to continuously reduce its loss function during the training process, and the loss function is represented as: The reward function r in the loss function is represented as: Each agent can find the optimal task allocation strategy b by continuous training a And all the agents' b a Combination becomes the task allocation strategy set B of the whole agent cluster.

3. The hierarchical reinforcement learning method for auxiliary tasks under uncertainty in adversarial scenarios according to claim 1, wherein: The agent obtains its task allocation strategy b a Finally, all entities, including agents and tasks, are recorded as E a , that is, E a =a∪b a , and then get E through the alpha shapes algorithm a Contour points, record this contour point set as L a ; In order to expand the sub-environment, L a Each element l in a , find the closest one that does not belong to E a Other points e′ in the environment a , and calculate l a The minimum distance from the map border, if this minimum distance is less than l a to e′ a The distance is, then the point on the map boundary corresponding to this minimum distance is added to the set E a ', otherwise e' a Add to E a ′, if l a Near the map boundary, the map boundary point is used as the boundary point of the sub-environment; then the convex hull method is used to calculate E a ′, and record this contour point set as L a ′, and finally L a By connecting the points in ′ in sequence, we can get the boundary of the sub-environment.

4. The hierarchical reinforcement learning method for auxiliary tasks under uncertainty in adversarial scenarios according to claim 1, wherein: At each agent gets its exclusive sub-environment z a After that, since each agent's sub-environment is different, the size of each agent's observation space is different, and at the same time for a certain agent, its sub-environment will change with time, that is, its observation space size will dynamically change, therefore, the attention mechanism needs to be used to process this variable observation space; for each agent, it needs to constantly optimize its network parameters θ during training, that is: where τ represents a sequence of states and actions produced by the agent interacting with the sub-environment during training, π θ (τ) represents the probability of the sequence τ under the current network parameters θ, denoted as: t = s1, u1, s2, u2,..., s t t ​ The policy gradient of this layer of reinforcement learning is represented as: And use this gradient to update the network parameters θ, that is: During the training process, the reward function r in the loss function is updated incrementally every time step, which is specifically represented as: Through continuous training, each agent can finally find the optimal specific behavior u of executing the task under the current allocation strategy, that is, the moving path.

Images