Multi-agent reinforcement learning task planning and control method under space-time task driving

Abstract

The invention provides a multi-agent reinforcement learning task planning and control method under space-time task driving. The method comprises the following steps: S1, establishing a reinforcement learning controller training architecture of a signal sequential logic task under a multi-agent system; s2, establishing a self-attention action compensation mechanism among the multi-agent systems; s3, constructing a robust value filtering baseline corresponding to the signal sequential logic task according to the task framework; s4, according to the environment information, soft constraints are generated for intelligent agent actions; s5, constructing a conflict detection and task replanning mechanism of the multi-target task according to the compensation action degree; and S6, based on the constraint conditions, iteratively training controller parameters so as to provide a control law online to meet the STL task requirements. According to the method, the controller parameters are learned in a reinforcement learning mode, the method adapts to the online solution control law in a complex environment, the given control law better conforms to the actual conditions of the external environment and the internal dynamic system, and the burden of the online calculation control law is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent system task planning and controller training, and in particular to a multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks. Background Art

[0002] In recent years, with the development of automatic control theory and artificial intelligence, and the growing demand for mechanization in the industrial field, multi-agent systems have received more and more attention. A multi-agent system is a computing system composed of multiple independent agents that can perceive the environment, communicate with each other, cooperate or compete to solve complex problems and perform tasks. Due to the advantages of multi-agent systems in terms of economy, safety, efficiency, etc., its application prospects are becoming more and more extensive. In the military field, multi-agent systems can perform tasks such as reconnaissance, attacking enemy targets, and logistics support; in the civilian field, multi-agent systems can be used for dangerous tasks such as rescue of forest and urban fires; in the industrial field, the division of tasks and collaborative operation of automated production lines in intelligent manufacturing, and the transportation and resource scheduling of goods in intelligent logistics all require the collaboration of multi-agent systems. With the continuous expansion of the application scenarios of agents, the tasks they face show complex temporal logic characteristics, which not only include the physical location constraints of dynamic systems, but also include time and logical requirements such as task time limits and execution order. The online decision-making and controller design of such complex temporal logic tasks face huge challenges, and formalized temporal logic tasks need to be introduced.

[0003] However, as the unknown nature of system dynamics increases and environmental changes become more complex, model-based methods have become increasingly challenging in solving agent decision-making problems. In the face of these uncertainties and complexities, reinforcement learning (RL) has attracted much attention in the field of agent control in recent years. Compared with traditional model-based methods, reinforcement learning uses trial and error to learn the optimal decision-making strategy through the interaction between the agent and the environment. This learning method makes reinforcement learning more suitable for situations where system models are difficult to model or difficult to obtain. Due to RL's excellent real-time decision-making performance, it is widely used in sequential decision-making problems that require a series of decisions to be made at different time steps. The concept of STL is introduced in the field of reinforcement learning. By giving full play to the learning ability of deep reinforcement learning and the formal representation of the temporal logic framework, the system tasks and performance specifications can be described more accurately, thereby effectively guiding the deep learning model to learn strategies that are more in line with practical requirements. The combination of this formal method and deep reinforcement learning provides new possibilities for solving multi-objective temporal logic task constraints. Especially in the context of increasing demand for accurate modeling and verification of system behavior, this has opened up new research directions for dealing with agent control problems with formal requirements and promoted further development in the field of agent control.

[0004] In the framework of multi-agent reinforcement learning, each agent receives its own temporal logic task of one or more goals. Since the tasks between agents are coupled, whether an agent can meet the task will also be affected by the behavior trajectory of neighboring agents. For example, in warehousing and logistics scenarios, under the premise of ensuring safety, the agent needs to plan the execution order of each task according to its own multi-objective time-limited material transportation task in order to complete multiple arrival tasks within a given time. In logistics warehousing scenarios with dense obstacles, whether the controller performance of a single agent can meet the requirements of temporal logic is not only affected by the order of material transportation, but also by the position status of neighboring agents. This type of multi-target point task with conflicts between agents is reduced to a complex multi-objective temporal logic task. The mutual influence of the states between the above agents and the impact of task planning on the satisfaction of temporal logic constraints will lead to poor training performance of the agent controller.

[0005] Therefore, it is crucial to solve complex temporal logic task decision-making problems with multiple target points, make full use of the experience of intelligent agents and achieve collaboration between intelligent agents. Designing a multi-agent reinforcement learning controller training strategy suitable for complex environments, which can enhance the controller's adaptability to dynamic environments and improve the controller's control performance, is of great significance for the realization of multi-agent systems with multi-target temporal logic constraints.

[0006] At present, no description or report of similar technology to the present invention has been found, and similar information at home and abroad has not been collected yet. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks. The method provides a reinforcement learning task planning and control algorithm that gives the multi-agent controller perception capabilities and ensures the completion of all tasks in an obstacle environment with collision avoidance conflicts.

[0008] According to one aspect of the present invention, a multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks is provided, comprising:

[0009] S1, establish the reinforcement learning controller training framework for signal timing logic tasks in multi-agent systems;

[0010] S2, establish a self-attention action compensation mechanism among multi-agent systems;

[0011] S3, construct the robust value filtering baseline corresponding to the signal timing logic task according to the task framework;

[0012] S4, generates soft constraints on the agent's actions based on environmental information;

[0013] S5, constructs the conflict detection and task replanning mechanism of multi-objective tasks according to the degree of compensation action;

[0014] S6, based on the constraints of S2\S3\S4\S5, iteratively train the controller parameters of S1 to give the control law online to meet the STL task requirements.

[0015] Preferably, S1, establishing a reinforcement learning controller training architecture for signal timing logic tasks in a multi-agent system, comprising:

[0016] S1.1, establish a multi-agent state space model:

[0017] Consider n agents i, The multi-agent system composed of the agent dynamic system is described as:

[0018]

[0019] in, and Represents the horizontal and vertical position information and control input information of agent i respectively; the state and control input of the agent are bounded and satisfy the constraints as well as

[0020] S1.2, based on the multi-agent state space model, a multi-agent temporal logic task normalization framework is established:

[0021] Based on the STL formal statements, the multi-agent signal timing logic tasks are standardized and the mathematical model of the task scenario is established:

[0022]

[0023] Where φ′ and φ″ are STL formulas, in the form of equation φ; the time interval [a, b] represents a series of time steps [a, a+1, …, b]; and ∧ are the logical operators "NO" and "AND"; G, F, and U are the temporal operators "global", "final", and "until"; for a discrete-time signal x, Indicates that the signal x at discrete time t satisfies the constraint φ; Indicates that the constraint φ is not satisfied; It means that φ′ and φ″ are satisfied at the same time; Indicates that the constraint φ must always be satisfied within the time interval [a,b]; Indicates that the constraint φ must be satisfied at least at one time in the future time interval [a, b]; operator It means that the constraint φ″ must be satisfied at least once in the time interval [a, b], and the constraint φ′ must be continuously satisfied before the constraint φ″ is satisfied; STL is based on multiple predicate logics μ, each of which consists of a column defined in the system state space. The predicate function in the actual physical meaning Decision; The condition h(x) ≥ 0 is used to mathematically describe that the actual system is in the required set; the predicate μ is defined as follows:

[0024]

[0025] STL provides a robust semantics that defines a real-valued function ρ φ , used to indicate that if ρ φ (x,t)≥0, then x satisfies φ, and the degree to which the signal x satisfies or violates φ is recorded;

[0026] The robust semantics is defined as:

[0027]

[0028] Among them, the real-valued function ρ φ It is used to describe the degree to which the system trajectory satisfies a specific STL constraint and is expressed as a real value with robust semantics; the positive and negative signs indicate the degree to which the system state trajectory violates or satisfies the constraint; in the mathematical model of the task scenario, when two constraints intersect, the minimum value of the robustness should be taken to ensure that they can be continuously satisfied at the same time; when two constraints are combined, the maximum value of the robustness of the two constraints is taken to ensure that both can be satisfied once; when with the global operator G, the minimum value of the robustness of the two constraints is taken; when with the final operator F, the maximum value of the robustness is taken; when with the until operator U, the time node needs to be introduced At this time point, the system state trajectory satisfies both constraints φ″ and φ′, that is, and When ρ is satisfied φ When (x,t)≥0, it means

[0029] S1.3, construct multi-agent temporal logic task rules:

[0030] The signal timing logic task of agent i is:

[0031]

[0032] in, are the horizontal and vertical spatial positions of agent i at time t; Indicates that agent i needs to be in the target area r i The task is expressed as a predicate function:

[0033]

[0034] S1.4 Establish a multi-agent reinforcement learning MARL framework:

[0035] Formulate multi-agent signal temporal logic tasks as Markov games Includes several states and actions Each action can transform the current state into a new state and obtain a reward;

[0036] N: the set of agents in the Markov game; denote all agents as Where N is the total number of agents;

[0037] represents the joint state set of all agents; represents the spatial position of agent i at time step t, Represents the joint state of N agents;

[0038] The joint set of actions of all agents; represents the actions taken by agent i in two directions at time step t, represents the joint action of N agents;

[0039] State transition probability function; where P(s t+1 =s′|s t =s,a t =a) maps the state s t+1 In status t Take action a t The state distribution after that, P:S×A×S′→[0,1];

[0040] Reward function; Indicates that in state s t Take action when The immediate reward generated later;

[0041] Setting: The goal of each agent i is to learn a parameter θ i Strategy π i (s t θ i ) to maximize their cumulative reward; the interactions between agents are through their actions This is achieved by considering the joint actions of each agent; these actions will affect the state of the environment s. t And each agent receives a reward signal r i ;

[0042] Under the signal timing logic STL task, a training framework for multiple agents with centralized training and decentralized execution is established. Consider a game with N agents, whose game space is shaped like a tuple

[0043] The policy network parameters of the agent are θ=(θ1,…,θ N ), and define the set π={π1,…,π N} is the strategy network of all agents;

[0044] The expected evaluation function corresponding to the policy network action of agent i is defined as Then the parameter θ i The expected gradient of is written as:

[0045]

[0046] in, is to take into account all the action information A of the other agents t The joint value function of agent i, s t is the state information of each agent;

[0047] Value Function It can measure the STL robustness value of the agent at the current state s;

[0048] Each They are all learned separately, and the agents have independent reward function structures, including collision avoidance penalties in static environments;

[0049] The solution to the policy gradient is based on the experience replay pool The form of taking the mean after sampling is:

[0050]

[0051] Among them, π i Similar to the policy network in DDPG to give continuous action values; experience replay pool Contains the state-action-reward-new state tuple This tuple is called an experience, and The experience of all agents is recorded;

[0052] Based on this experience replay pool Centralized action-value function for multiple agents The parameter update rules are as follows:

[0053]

[0054] Among them, π={π′1,…,π′ N} is a function with target value parameters θi i The target strategy set;

[0055] S1.5, establish the multi-agent reward function design rules under the STL specification:

[0056] For each agent, the RT of the STL task reaches the task part F [a,b] G [0,τ] , only considering its robust value within τ time steps;

[0057] The historical state of τ time steps is expressed as And by inputting s into the policy network t-τ:t And get action a t , and then execute the action into the agent dynamic system to obtain the state s at the new moment t+1 ;

[0058] At time t+1, the τ-step history state will be updated and the corresponding RT reward R will be generated i (s t-τ:t ,t), which is defined as:

[0059]

[0060] For the collision avoidance task and obstacle avoidance task, the reward function is described as:

[0061]

[0062]

[0063] Among them, R i (s t-τ:t ,t) CAT Usually represents the closest distance between the current agent i and other agents, which is greater than d safe It is safe;

[0064] Based on the components of these reward functions, the reward function of any agent is obtained as:

[0065] R i (s t-τ:t ,t)=α*R i (t) RT +β*R i (t) OAT +γ*Ri (t) CAT ,(12)

[0066] Among them, α, β, and γ are the weight hyperparameters of the rewards for each part of the STL task.

[0067] Preferably, S2, establishing a self-attention action compensation mechanism among multi-agent systems, including:

[0068] S2.1, preprocessing future time robust trajectory data:

[0069] Based on the motivation of utilizing future robust information, the data of the experience replay pool is expanded to Contains tuples in, represents the robust value of the time-limited arrival task at time t, represents the robustness value of the collision avoidance task part of the STL task of agent i;

[0070] use To describe The capacity of the robust value in , where represents the complete robust trajectory from 0 to T The number of,T represents the entire time domain of the STL task;

[0071] 2.2, construct a robust trajectory self-attention mechanism module:

[0072] Based on the information sampled by multiple agents at time k A self-attention mechanism based on robust value trajectories is designed to extract the influence relationship between agents. The weight generation network of the self-attention module is:

[0073]

[0074] in, represents the impact of the robust information of agent j on agent i;

[0075] S2.3, construct the self-attention network parameter update rules during training:

[0076] At each training moment, a compensation time k∈[0,T] is first randomly selected;

[0077] By making the sum of the cumulative rewards of the trajectory generated after compensation greater than the reward before compensation, the compensation network parameter θ is designed. c ,W K W Q and W V The update rule is:

[0078]

[0079] Among them, V c Is the agent accepting compensation action The maximum cumulative reward value that can be achieved after , is recorded as the compensation value function, is the state and action data of the same sample in the robust data extraction process, It means that only the compensation action is performed on agent i in all action spaces. Agent joint action, expressed as Compared with Only the i-th element is replaced; the regularization part in formula (22) makes the compensation action as small as possible to make the training more stable;

[0080] Add constraints to the update rule:

[0081]

[0082]

[0083] Among them, V is the original value function, and the trainable parameter is θ q Formula (24) represents the experience pool and The sampled data has the same index position in the experience pool, that is, the state-action-reward experience and robust value experience generated at the same time in different rounds of the training process.

[0084] At the same time, update π according to the following gradient ascent method i Parameters:

[0085]

[0086] The parameters of both value functions are updated using the objective function, and the update rules are as follows:

[0087]

[0088] Among them, V c ′ and V′ are the two objective value functions of the compensation value function and the original value function, and their parameters are θ′ q,c and θ′ q ; Update these parameters as follows based on the learning rate lr and the target value function update factor α:

[0089]

[0090] Preferably, in step S2.1, use To describe The capacity of the medium robust value is:

[0091] Randomly select a timestamp k and calculate the batch size according to the batch size. ρ Extract multiple robust value sequences to obtain a data set as the input of Shape; the data set is:

[0092]

[0093] in, The data structure is that for the part of agent i from time k to T, S = Size ρ Strip robust trajectory;

[0094] Sample multiple collision avoidance information trajectories:

[0095]

[0096] Among them, Size Col =Size ρ Indicates the number of samples extracted;

[0097] Combine each element of these two inputs into The following robust information input is obtained:

[0098]

[0099] The self-attention model in S2.2 implements an approximate representation of the influence weight, specifically:

[0100] The query vector of agent i at time step t is expressed as in, Size ρ A random sample [:] index means extracting the corresponding fragment information of all agents, W Q is the trainable parameter matrix in the model;

[0101] The key vector and value vector information of agent i and agent j at time step t are respectively expressed as and

[0102] The calculation method of the self-attention influence weight between agents is:

[0103]

[0104] Among them, d represents the dimension of parameter matrices Q and K, and weights is the influence factor between agent i and agent j at time t;

[0105] Based on the influence factor of agent k In the case of , the action compensation coefficient that agent i should make for:

[0106]

[0107] Among them, ⊙ is the bitwise product of the matrix; the physical meaning of the compensation factor is that if the input of j The future robust value after the k moment represented by must meet the requirements of the STL specification. At this time, agent i should additionally execute the original strategy π i Some compensatory actions other than

[0108] For agent i, in the combination of different agents Compensation factor Afterwards, the final compensation factor at sampling time k is given as:

[0109]

[0110] in, represents the compensation response that agent i should make at time k;

[0111] Normalize the final compensation factor:

[0112]

[0113] Among them, MLP is a fully connected layer neural network. is the compensation action network MLP of agent i i At the same time, the activation function tanh is used to limit the size of the compensation action to prevent the gradient explosion during training;

[0114] Applying compensation actions directly to the agent’s original policy network The synthesized action is expressed as:

[0115]

[0116] Preferably, S3, constructing a robust value filtering baseline corresponding to the STL task according to the task framework, including:

[0117] S3.1, Design of robust value filtering baseline:

[0118] Subtract the actual robust value from the baseline to obtain the filtered robust value information:

[0119]

[0120] Among them, B(k) represents the baseline value of the robust filter at time k, that is, the expected robust value of the [0,T] time domain series at that time, and k:T represents the interception of records between time k and T of each complete record;

[0121] S3.2, design rules for filtering robust information:

[0122]

[0123] If the robust trajectory is evenly distributed around the baseline and above the baseline, we get The compensation action network will not be activated and no compensation action will occur.

[0124] Preferably, the form and parameters of the robust value filtering baseline in S3 are restricted so that the robust data after baseline filtering can fully express the satisfaction of the task at each control moment, specifically:

[0125]

[0126]

[0127] ρ max ≥Baseline(T)≥0, (30-c)

[0128] Update the baseline parameters by gradient ascent:

[0129]

[0130] When the parameters of the Baseline converge and the robust trajectory is above the Baseline, at time T, the robust value is greater than 0, indicating that the signal timing logic task of agent i is satisfied.

[0131] Preferably, S4, generating soft constraints on the agent's actions according to the environmental information, includes:

[0132] S4.1, construct the positive direction of the auxiliary direction coordinate system:

[0133] Revisiting the pool based on experience Extracted from t information, and obs , will s obs As the origin in space, As the positive direction of the x-axis in this space, e x ;

[0134] S4.2, generate auxiliary directions:

[0135] In setting the positive direction e x Afterwards, from Size of the sample train The tuple is divided into two parts: s t,II,III and t,I,IV , representing the second and third quadrants and the first and fourth quadrants of the sampled data respectively; these two parts of data correspond to the capacity Size II,III and Size I,IV ;

[0136] S4.3 calculates the following reference direction vector in the sample based on the agent target point and agent position information:

[0137]

[0138]

[0139] Wherein formula (32-a) represents the target direction in the RT task, and formula (32-b) represents the collision direction in the OAT task.

[0140] S4.4, determine a collision-avoiding direction e based on the two environmental information of the reference direction vector obstacle,⊥ , the constraints of each element in this direction are:

[0141] (e obstacle，⊥ ) T e obstacle 0, (33-a)

[0142] ||e obstacle，⊥ ||=||e target ||, (33-b)

[0143]

[0144] For states in quadrants I and IV t,I,IV , their final auxiliary directions are as follows:

[0145]

[0146] And the two auxiliary directions are applied to the policy network update using the following update rules:

[0147]

[0148] where arccos(·,·) represents the angular overlap between the action and the auxiliary direction obtained by π; the hyperparameter α a The larger it is, the less dependent the policy network is on the auxiliary directions.

[0149] Preferably, S5, constructing a conflict detection and task replanning mechanism for multi-objective tasks according to the degree of compensation action, including:

[0150] S5.1, Constructing multi-objective STL task specifications:

[0151] The tasks assigned to each agent i include: visiting CT in the time domain [0,T] i public target area, and stay in each area for τ seconds; visit PT in the time domain [0,T] i private target areas, and stay for τ seconds respectively; keep a safe distance d from obstacles obs ; Keep a safe distance d from other agents col ;

[0152] The multi-objective STL task is expressed as:

[0153]

[0154] S5.2, controller learning layer optimization:

[0155] Under the training framework, the value function is the cumulative reward of all agents When the target point of an agent i is switched, its cumulative reward will also change, that is,

[0156]

[0157] Among them, k i represents the target point of agent i at the current moment. When the task k of agent i at time t is i Completed and switched to k i ′≠k i,t , at this time, due to task k i ′ has not yet been completed, so the agent reward at time t+1 is Make That is, switching targets leads to a decrease in the cumulative estimated reward;

[0158] Embed the value network q with a one-hot vector e i ,i∈{1,2,…,N}, to represent the current The set of priorities of all agents evaluated at the current moment

[0159] rank t =(rank t，i |i∈{1,2,…,N}), (38)

[0160] Among them, the execution order of all subtasks of agent i is from index 1 to N i rank t,i Sure,

[0161] Assign agent i to task k at time ti The one-hot vector of is represented using a fixed-length form:

[0162]

[0163] Among them, splitFactor is a positive integer hyperparameter, which represents the discrimination of the task encoding vector;

[0164] Encode the task that the agent is processing at the current time t Add to the experience replay pool to form a state-action-reward-hot vector experience replay pool:

[0165]

[0166] Get the final value function and policy network optimization target:

[0167]

[0168] in, represents the goal k of agent i i The corresponding vector, its policy network Input a vector containing only i θ q ,θ′ q Represent the network parameters of the value function and the target value function respectively, Represents the policy network π i Network parameters;

[0169] When agent i reaches target point k at time t i to k i ′, the collection rule of robust value reward is defined as follows:

[0170]

[0171] When task k i Completed, its subsequent cumulative robust reward value will remain At this time, task k i The cumulative robust value of is not easy to change to below k i ′, so that k i Priorities are more stable;

[0172] S5.3, construct a priority network:

[0173] Each agent i reads each subtask The cumulative robust value vector from the controller learning layer:

[0174]

[0175] in, Indicates that agent i has The cumulative robustness value of a task in the time domain [0,T].

[0176] For each agent i’s multi-objective task that can be characterized by the STL specification, its priority network is expressed as a probability output network as follows:

[0177]

[0178] Among them, N i =GT i +PT i represents the sum of public and private access tasks assigned to agent i in the task time domain [0, T]. Before each round of training, the experience pool of all task robust trajectories of each agent i collected in the previous training rounds is replayed Extract an experience from it and calculate each target point k i Cumulative Robust Vector The accumulated robust value vector is input into the probability output network to obtain the priority probability Determine the task priority rank based on the order of probability from large to small i , and then input into the controller learning layer to guide the controller according to the rank i Switch and train the encoding vector;

[0179] S5.4, Robust trajectory data processing:

[0180] S5.4.1, extract time nodes to detect collision:

[0181] In the time domain T, first extract the detection time point t check , denoted as t c , and a small time slice [T start ,T end ], the fragment must meet certain requirements,

[0182] t c ∈[T start ,T end ], (45-a)

[0183] |T start -T end |≥τ, (45-b)

[0184] T end ≤T, (45-c)

[0185] Subtask The intercepted batch robust trajectory is expressed as follows:

[0186]

[0187] Where S = Size ρ Indicates the batch size of the extracted fragments. For each row of sample data Calculate whether it has suffered a conflict and detect whether the exploration efficiency is low, that is, whether the average rising rate of the accumulated robust value in the slice meets the requirements; the slice robust change rate is as follows:

[0188]

[0189] Among them, speed is the maximum limit of the agent's control law;

[0190] S5.4.2, detect the amount of conflict at the sampling time point:

[0191] A robust baseline is designed for each agent to meet different robustness value change requirements; multiple To perform different robust value filtering on different subtasks, in order to reduce the probability of individual subtasks being judged as conflicts due to their robust values ​​being lower than the baseline values, the following baseline cluster is constructed:

[0192]

[0193] According to the task k of agent i at the current moment i , to increase the sampling sample accordingly, k i Baseline The batch size of the samples used to prevent baselines for other subtasks and task k i ′’s robust trajectory is overfitted to reduce the misjudgment of conflicts;

[0194] S5.4.3, Re-prioritization based on conflicting values:

[0195] Calculate the performance of each agent i in their respective task k at random time k i Compensation action The scaling factor of the task is updated according to the size of the compensation action, and then the slice robust trajectory (46) is scaled accordingly;

[0196]

[0197] Among them, the scaling factor Measure the task k performed by agent i i The number and size of the compensation action. The scaling rules (49-c) and (49-e) indicate that only the task k that is in progress at time k iThe robust value of the fragment is scaled, and the robust values ​​of the remaining subtasks remain unchanged; at the same time, the robust values ​​of the remaining agents in conflict are scaled as shown in formula (49-f), where represents the set of agents that visit the same area as i and also suffer from conflicts; at this time, in a sampling sample, for all N agents i i The cumulative robust value of a subtask in the time domain [0, T] is calculated as follows:

[0198]

[0199] And the priority network parameters of each agent i are updated using the following cumulative robust weighted cross entropy objective function:

[0200]

[0201] in, It means that the robust trajectory of each task of agent i is sampled in Monte Carlo in the experience replay pool, and the average is taken to obtain the target loss function value after calculating the value.

[0202] Preferably, S6, iteratively training controller parameters to provide a control law online to meet the STL task requirements, including:

[0203] S6.1, Initialization: Initialize all network parameters, initial position of intelligent agents, multi-intelligence multi-objective STL task form, and hyperparameter settings in the network;

[0204] S6.2, collect and update the experience revisit pool: take each execution of the policy network T times as a training cycle, collect the position information, action information, rewards, and task encoding of T agents as a group, and when the amount of data exceeds the capacity of the experience revisit pool, overwrite the data at the corresponding time in the experience revisit pool from the beginning;

[0205] S6.3, continuously switch task encoding in each training cycle: according to the priority of each agent in the current cycle, when task k i After completion, switch to the next task k i ′, and at the same time the encoding vector Update the input of the value network and the policy network, and use the gradient descent method to update all networks;

[0206] S6.3, continuously update the priority: each cycle is executed, the task priority of each agent in the next cycle is determined according to the probability distribution of the output after the cumulative robust value is input to the priority network; at the same time, the priority network parameters are updated; until in P cycles, the position trajectory of all agents from time 0 to time T in each cycle meets the STL task specification, or the MAX is terminated. epochAfter training for cycles, the training is terminated and the policy network parameters of the final N agents are obtained;

[0207] S6.4, Test Strategy Network Performance: In P T In a test cycle, an initial position is selected for each agent, and the control law is continuously output according to the position information from time 0 to T, which is applied to the agent dynamic system to obtain the agent position trajectory of length T; and the position trajectory of each agent is tested to determine whether it meets the STL specification and P T The number of P tests that meet the requirements success .

[0208] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0209] 1. The embodiment of the present invention uses reinforcement learning to learn the parameters of the controller to adapt to the problem of difficulty in solving the control law online in a complex environment. The given control law is more in line with the actual situation of the external environment and the internal dynamic system, and greatly reduces the burden of online calculation of the control law.

[0210] 2. The embodiment of the present invention is applicable to the conflict scenario where the initial position of the intelligent agent is random and each intelligent agent performs multiple time-limited arrival tasks in the same time period, so that all time-limited tasks can be completed while meeting the requirements of collision avoidance and obstacle avoidance.

[0211] 3. The embodiment of the present invention provides a training method for compensating the output of the intelligent agent strategy network using a robust attention mechanism, thereby giving intelligent agents the ability to perceive conflicts, respond to conflicts, and collaboratively complete time-limited arrival tasks.

[0212] 4. An embodiment of the present invention provides a priority update strategy for re-planning the priority of multi-objective tasks of each intelligent agent, so that multiple agents can explore the environment and train controllers more efficiently, thereby meeting the requirements of temporal logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0213] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0214] Figure 1 is a flow chart of a multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks in an embodiment of the present invention;

[0215] Figure 2 It is a multi-agent multi-objective STL task simulation environment diagram in an embodiment of the present invention;

[0216] Figure 3 is a motion trajectory diagram of an intelligent agent without adding a detection mechanism in an embodiment of the present invention;

[0217] Figure 4 is a motion trajectory diagram of an intelligent agent with a detection mechanism added in an embodiment of the present invention;

[0218] Figure 5 is a cumulative reward curve diagram of agent A in an embodiment of the present invention without adding a detection mechanism;

[0219] Figure 6 is a cumulative reward curve diagram of agent B in an embodiment of the present invention without adding a detection mechanism;

[0220] Figure 7 is a cumulative reward curve diagram of agent A joining the detection mechanism in an embodiment of the present invention;

[0221] Figure 8 is a cumulative reward curve diagram of agent B in an embodiment of the present invention without adding a detection mechanism; DETAILED DESCRIPTION

[0222] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0223] The present invention provides a method for multi-agent reinforcement learning task planning and control driven by spatiotemporal tasks. The method is aimed at the following multi-agent multi-objective STL task planning and control problems and is implemented based on the following principles.

[0224] The task planning and control problem of multi-agent system based on reinforcement learning under multi-objective temporal logic tasks can be described as:

[0225] Taking the task allocation of warehouse logistics as an example, there are multiple known obstacle points in the known space, and a multi-agent system composed of N unmanned transportation agents performs time-limited arrival and transportation tasks in the area. The tasks assigned to each agent i include: (1) visiting CT in the time domain [0, T] i public target area, and stay in each area for τ seconds; (2) visit PT in the time domain [0,T] i private target areas, and stay in each area for τ seconds; (3) keep a safe distance d from obstacles obs ; (4) Maintain a safe distance d from other agents col Specifically, we can consider N = 2 agents, whose agent dynamic equation is as follows:

[0226]

[0227] In the above expression, represents the position state of the agent at time step t, where and are the horizontal and vertical coordinates of the agent respectively. represents the control input of the agent at time step t, are the control inputs of the agent on the x and y axes respectively. It indicates that the environmental interference term follows a zero-mean normal distribution with a variance of 0.1.

[0228] The multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks provided by the example of the present invention is conceived as follows:

[0229] In the offline state, according to the given multi-objective STL task, the policy network, value network, reward function, robust filtering baseline, self-attention network, auxiliary direction, priority network and conflict detection mechanism parameters are initialized for each agent. For each agent, a square area of ​​the initial position is given. Each agent randomly selects the initial position. In a cycle, the corresponding information in the experience replay pool is continuously collected. At each moment, the parameters of each network are updated based on the given policy network and compensation mechanism. During this period, the task encoding vector of the agent is continuously updated according to the task completion status. After each T time step of training, the robust filtering baseline is updated again, and the task priority and priority network used in the next cycle are updated according to the conflict detection mechanism. It is iterated until the STL tasks of all agents are met in P cycles. Finally, the policy network parameters of each agent and the task priority of each agent are obtained.

[0230] During online execution, each agent still randomly selects its own starting point in the initial area, and inputs the multi-agent state and task code into the policy network at each moment to obtain the agent's control law for the next moment, which is applied to the agent dynamic system to obtain the agent state at the next moment, and switches the task code or keeps the code unchanged according to the completion status of multiple time-limited tasks and task priorities, and finally obtains the motion trajectory of each agent within T time steps.

[0231] Based on the above planning problems and principles, the embodiment of the present invention provides a multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks, and the specific process is as follows:

[0232] S1, establish the reinforcement learning controller training framework for signal timing logic tasks in multi-agent systems;

[0233] S2, establish a self-attention action compensation mechanism among multi-agent systems;

[0234] S3, construct the robust value filtering baseline corresponding to the signal timing logic task according to the task framework;

[0235] S4, generates soft constraints on the agent’s actions based on environmental information;

[0236] S5, constructs the conflict detection and task replanning mechanism of multi-objective tasks according to the degree of compensation action;

[0237] S6, based on the constraints of S2\S3\S4\S5, iteratively train the controller parameters of S1 to give the control law online to meet the STL task requirements.

[0238] In order to handle complex tasks and interactions in a multi-agent system and improve system performance, a preferred solution of step S1 is provided in a preferred embodiment of the present invention, wherein S1 establishes a reinforcement learning controller training framework for a signal timing logic (STL) task in a multi-agent system, including:

[0239] S1.1, establish a multi-agent state space model:

[0240] Consider n agents i, The multi-agent system consists of agents, and the agents can be described by the following dynamic system:

[0241]

[0242] in, and Represents the horizontal and vertical position information and control input information of agent i respectively. The state and control input of the agent are bounded and satisfy certain constraints as well as In a multi-agent multi-objective task, each agent will be independently assigned multiple sequential logic subtasks. Since the target points between agents may conflict, whether a single agent can complete all tasks depends on the choice of target areas by the remaining agents.

[0243] S1.2, establish a multi-agent STL task standardization framework:

[0244] Based on the STL formal statements, the temporal logic tasks of the multi-agent system are described in a standardized manner, and the mathematical model of the task scenario is established:

[0245]

[0246] where φ′ and φ″ are STL formulas of the form φ. The time interval [a,b] is described as a series of time steps [a,a+1,…,b]. and ∧ are the logical operators "not" and "and". G, F, and U are the temporal operators "global", "final", and "until". For a discrete-time signal x, It means that the signal x at discrete time t satisfies the constraint φ. Indicates that the constraint φ is not satisfied; It means that φ′ and φ″ are satisfied at the same time; Indicates that the constraint φ must always be satisfied within the time interval [a,b]; Indicates that the constraint φ must be satisfied at least at one time in the future time interval [a, b]; operator It means that the constraint φ″ must be satisfied at least once in the time interval [a, v], and the constraint φ′ must be continuously satisfied before the constraint φ″ is satisfied. STL is based on multiple predicate logics μ, each of which consists of a column defined in the system state space. The predicate function in the actual physical meaning The condition h(x)≥0 is used to mathematically describe that the actual system is in the required set. The predicate μ is defined as follows:

[0247]

[0248] Considering that the multi-unmanned system in actual scenarios will be affected by external temperature, noise, electromagnetic interference and internal dynamic structure errors, unpredictable accidents may occur in the face of complex mission scenarios, such as obstacle avoidance, collision avoidance, information interaction, etc., resulting in low mission completion efficiency or increased engineering losses. Therefore, when designing the controller of the multi-unmanned system, sufficient margin needs to be left to maximize the mission completion. STL provides a robust semantics, which defines a real-valued function ρ φ , used to indicate that if ρ φ (x, t) ≥ 0, then x satisfies φ, and the degree to which the signal x satisfies or violates φ is recorded. The robust semantics is defined as follows:

[0249]

[0250] Among them, the real-valued function ρ φIt is used to describe the degree to which the system trajectory satisfies a specific STL constraint and is expressed as a real value with robust semantics. The positive and negative signs indicate the degree to which the system state trajectory violates or satisfies the constraint. In formula (2), when two constraints intersect, the minimum value of robustness should be taken to ensure that they can be continuously satisfied at the same time; when two constraints are combined, the maximum value of the robustness of the two constraints is taken to ensure that both can be satisfied once; when there is a global operator G, the minimum value of the robustness of the two constraints is taken; when there is a final operator F, the maximum value of robustness is taken; when there is an until operator U, a time node needs to be introduced At this time point, the system state trajectory satisfies both constraints φ″ and φ′, that is, and In the above expression, when ρ φ When (x,t)≥0, this means

[0251] S1.3, Multi-agent STL task construction rules:

[0252] Consider the following STL task for agent i:

[0253]

[0254] in, are the horizontal and vertical spatial positions of agent i at time t. Indicates that agent i needs to be in the target area r i The task can be expressed by the following predicate function:

[0255]

[0256] S1.4, Establish a multi-agent reinforcement learning (MARL) framework:

[0257] In MARL, all agents need to jointly meet the arrival and collision avoidance requirements of STL. The multi-agent STL task is formulated as a Markov game. It consists of a sequence of states and actions. Each action transforms the current state into a new state and obtains a reward. A Markov game can be represented by a tuple To describe, where:

[0258] That is, the set of agents in the Markov game. We denote all agents as Where N is the total number of agents.

[0259] Represents the joint state set of all agents. represents the spatial position of agent i at time step t, Represents the joint state of N agents.

[0260] The joint set of actions of all agents. represents the actions taken by agent i in two directions at time step t, Represents the joint action of N agents.

[0261] State transition probability function. Where P(s t+1 =s′|s t =s,a t =a) maps the state s t+1 In status t Take action a t The state distribution after that. Therefore P:S×A×S′→[0,1].

[0262] Reward function. Indicates that in state s t Take action when Instant rewards generated later.

[0263] The goal of each agent i is to learn a parameter θ i Strategy π i (s t θ i ) to maximize their cumulative reward. The interactions between agents are through their actions To achieve this, the policy network needs to comprehensively consider the joint actions of each agent. These actions may affect the state of the environment s t And each agent receives a reward signal r i .

[0264] Considering the complexity of multi-agent reinforcement learning compared to single-agent reinforcement learning, using a reward function that maps a single agent state to a real number to update the value function of each agent cannot fully utilize the information from other agents, especially when time information is not considered, the convergence of cumulative rewards will be unstable. This step considers establishing a centralized training and decentralized execution (CTDE) training framework for multiple agents under the STL task. Consider a game with N agents, whose game space is shaped like a tuple The policy network parameters of the agent are θ=(θ1,…,θ N ), and define the set π={π1,…,π N} is the policy network of all agents. In this section, the expected evaluation function corresponding to the policy network action of agent i is defined as Then the parameter θ i The expected gradient of can be written as:

[0265]

[0266] in, is to take into account all the action information a of the other agents t The joint value function of agent i, s t is the state information of each agent. In the algorithm of this chapter, the value function It usually measures the STL robustness value of the agent at the current state s. are learned separately, so the agent can have an independent reward function structure, including collision avoidance penalties in static environments, such as the collision avoidance task G [a,b] .

[0267] In the STL task considered by the present invention, since the continuous action space is considered, the solution of the policy gradient is adopted as follows based on the experience replay pool The form of taking the mean after sampling is:

[0268]

[0269] Among them, π i Similar to the policy network in DDPG to give continuous action values. Experience replay pool Contains the state-action-reward-new state tuple This tuple is called an experience, and Records the experience of all agents. Replays the pool based on this experience Centralized action-value function for multiple agents The parameter update rules are as follows:

[0270]

[0271] Among them, π={π′1,…,π′ N} is a function with target value parameter θ′ i The target policy set.

[0272] S1.5, establish the multi-agent reward function design rules under the STL specification:

[0273] For each agent's STL task in equation (4), the RT reaches the task part F [a,b] G [0,τ] , we only need to consider its robust value within τ time steps. The historical state of τ time steps is represented as st-τ:t = And by inputting s into the policy network t-τ:t And get action a t , and then execute the action into the dynamic model (1) to further obtain the state s at the new moment t+1 At time t+1, the τ-step history state will be updated and the corresponding RT reward R will be generated. i (s t-τ:t ,t), which is defined as:

[0274]

[0275] For the collision avoidance task and obstacle avoidance task, this section describes the reward function as:

[0276]

[0277] Among them, R i (s t-τ:t ,t) CAT Usually represents the closest distance between the current agent i and other agents, which is greater than d safe Based on the components of these reward functions, the reward function of any agent can be given as:

[0278] R i (s t_τ:t ,t)=α*R i (t) RT +β*R i (t) OAT +γ*R i (t) CAT ,(12)

[0279] Among them, α, β, and γ are the weight hyperparameters of the rewards for each part of the STL task.

[0280] Through the above embodiments, a reinforcement learning controller training framework for signal timing logic tasks in a multi-agent system is established, providing a simulation and learning environment for agents to optimize strategies through trial and error. By building this robust and efficient system, agents can autonomously plan and control tasks, adapt to environmental changes, and improve task completion rates.

[0281] Considering that the multi-agent system lacks a collaborative ability to perceive the degree of completion of each STL task when interacting with the environment, it is necessary to introduce a self-attention mechanism to extract the degree of conflict between agents and generate compensation actions to correct the actions output by the controller. Therefore, in a preferred embodiment, a preferred solution of step S2 is provided. Specifically, S2, a self-attention action compensation mechanism between multi-agent systems is established, including:

[0282] S2.1 Preprocessing future time robust trajectory data:

[0283] This step is based on the motivation of utilizing future robust information. The data of the experience replay pool first needs to be expanded to Contains tuples in, represents the robust value of the time-limited arrival task at time t, It represents the robust value of the collision avoidance task in equation (4). To describe The capacity of the robust value in , where represents the complete robust trajectory from 0 to T The number of, T represents the entire time domain of the STL task. And MADDPG training randomly sampled tuples Different empirical methods are used at different times in the process. This step will randomly select a timestamp k and calculate the batch size according to the batch size. ρ Extract multiple robust value sequences and obtain the following data set:

[0284]

[0285] in, The data structure is that for the part of agent i from time k to T, S = Size ρ At the same time, this step uses the same method to calculate multiple collision avoidance information trajectories (for simplicity, we use express ) for sampling, as follows:

[0286]

[0287] Among them, Size Col =Size ρ In order to integrate these two key information, each element of these two inputs needs to be combined into Thus, the following robust information input is obtained:

[0288]

[0289] S2.2, construct a robust trajectory self-attention mechanism module:

[0290] Consider the information sampled by multiple agents at time k In this step, a self-attention mechanism based on robust value trajectories is designed to extract the influence relationship between agents. The weight generation network of the self-attention module (Attention Model) is as follows:

[0291]

[0292] in, Represents the influence of the robust information of agent j on agent i. This section uses the self-attention model to achieve an approximate representation of this influence weight. The query vector of agent i at time step t is expressed as in, Size ρ A random sample [:] index means extracting the corresponding fragment information of all agents, W Q is the parameter matrix that can be trained in the model. At the same time, the key vector and value vector information of agent i and agent j at time step t are respectively expressed as and The calculation method of the self-attention influence weight between agents is designed as follows:

[0293]

[0294] Where d represents the dimension of the parameter matrices Q and K. Weight is the influence factor between agent i and agent j at time t. Consider the influence factor of agent j In the case of , the action compensation coefficient that agent i should make as follows:

[0295]

[0296] Where ⊙ is the bitwise product of the matrix. The physical meaning of this compensation factor is that if the input of j The future robust value after the k moment represented by must meet the requirements of the STL specification. At this time, agent i should additionally execute the original strategy π i For agent i, in the combination of different agents Compensation factor Afterwards, the final compensation factor at sampling time k is given as follows:

[0297]

[0298] in, represents the compensation response that agent i should make at time k. In order to meet the system dynamic constraints, such as speed constraints, this step normalizes the factor (19) as follows:

[0299]

[0300] Among them, MLP is a fully connected layer neural network. is the compensation action network MLP of agent i i Parameters. At the same time, the activation function tanh is used to limit the size of the compensation action to prevent the gradient explosion during training. The compensation action is directly applied to the original policy network of the agent. The synthesized actions are as follows:

[0301]

[0302] S2.3 Construct the self-attention network parameter update rules during training:

[0303] At each training moment, a compensation time k∈[0,T] is first randomly selected. By making the sum of the cumulative rewards of the trajectory generated after compensation greater than the reward before compensation, this step designs the compensation network parameter θ c ,W K W Q and W V The update rule is as follows (for simplicity, θ is used A Indicates parameter W K ,W Q and W V ):

[0304]

[0305] Among them, V c Is the agent accepting compensation action The maximum cumulative reward value that can be achieved after , is recorded as the compensation value function, is the state and action data of the same sample in the robust data extraction process. In particular, It means that only the compensation action is performed on agent i in all action spaces. Agent joint action, expressed as Compared with Only the i-th element is replaced. The regularization part in equation (22) makes the compensation action as small as possible to make the training more stable. At the same time, this step adds the following constraints when updating the target (22):

[0306]

[0307] Among them, V is the original value function, and the trainable parameter is θ q Formula (24) represents the experience pool and The sampled data of have the same index position in the experience pool, that is, the state-action-reward experience and robust value experience generated at the same time in different rounds of the training process. At the same time, update π according to the following gradient ascent method i Parameters:

[0308]

[0309] The parameters of the two value functions are updated using the objective function (8) and the update rules are as follows:

[0310]

[0311] Among them, V c ′ and V′ are the two objective value functions of the compensation value function and the original value function, and their parameters are θ′ q,c and θ′ q These parameters are updated as follows based on the learning rate lr and the target value function update factor α:

[0312]

[0313] The above embodiment designs a self-attention compensation network structure based on robust trajectories and its network update rules for the time-limited arrival tasks of all intelligent agents, which can effectively capture the key information in the robust trajectories collected from the experience replay pool, thereby generating compensation actions and correcting the controller output of the intelligent agent; improving the efficiency and safety of the collaborative control of the intelligent agents, and ensuring that the task is successfully completed within the limited time.

[0314] For the reuse of the robust value information generated during the training process, it is necessary to extract the robust value fragments that violate the task robustness requirements and use them to improve the training efficiency of the self-attention module. Therefore, in a preferred embodiment, a preferred solution of step S3 is provided. Specifically, S3 constructs a robust value filtering baseline corresponding to the STL task according to the task framework, including:

[0315] S3.1, Design of robust value filtering baseline:

[0316] Considering the unevenly distributed negative robust value trajectory, that is, the robust sequence value with large variance This will cause a large disturbance to the training and reduce the convergence of the compensation action network. Therefore, this step uses a robust reference trajectory Baseline to measure the satisfaction of the robust sequence at each moment, rather than waiting until time T to perform a global STL robust value measurement. Subtract the actual robust value from the baseline to obtain the filtered robust value information as follows:

[0317]

[0318] Among them, B(k) represents the robust filter baseline value at time k, that is, the expected [0,T] time domain series robust value at that time, and k:T means intercepting records between time k and T of each complete record.

[0319] S3.2, Design baseline parameter update rules:

[0320] In order to utilize the information that may contain collision avoidance, the part of the filtered robust information (28) that is less than 0 is considered, and the rule for filtering the robust information is designed as follows:

[0321]

[0322] If the robust trajectory is evenly distributed around the baseline and above the baseline, we get The compensation network (20) will not be activated and thus no compensation action will be taken. The same applies to filtering. If no collision occurs, the output will not be affected.

[0323] In particular, when the sampling size ρ robust trajectory (from sampling timestamp k to task time domain T), if the trajectory Uniformly located at the baseline of agent i i Above, it is determined that the trajectory meets the time logic task specification, and there is no need to be π i The generated policy adds compensation actions (20). In order to make the robust data after baseline filtering fully express the satisfaction of the task at each control moment, the form and parameters of the baseline need to be restricted as follows:

[0324]

[0325] ρ max ≥Baseline(T)≥0, (30-c)

[0326] Update the baseline parameters by gradient ascent as follows:

[0327]

[0328] When the parameters of the Baseline converge and the robust trajectory is above the Baseline, at time T, the robust value is greater than 0, which means that the STL specification (4) is met.

[0329] The above embodiment designs a robust filtering baseline for the STL task form of each time-limited arrival task, and designs a baseline parameter update rule to improve the ability to extract conflicting information in robust trajectory data, improve the task success rate, optimize the intelligent agent strategy, and ensure system security.

[0330] In order to ensure that the agent follows the temporal logic constraints, optimize its behavior in a complex environment, and improve the collaborative performance between multiple agents, in another embodiment of the present invention, a preferred solution of step S4 is provided, that is, S4, generating soft constraints for the agent actions according to the environmental information, specifically:

[0331] S4.1, construct the positive direction of the auxiliary direction coordinate system:

[0332] This step is based on the experience of the return visit pool Extracted from t Information, regardless of and obs How they are distributed in space will obs As the origin in space, As the positive direction of the x-axis in this space, e x .

[0333] S4.2, generate auxiliary direction:

[0334] In setting the positive direction e x After that, this step will start from Size of the sample train The tuple is divided into two parts: s t,II,III and t,I,IV , representing the second and third quadrants and the first and fourth quadrants of the sampled data. These two parts of data correspond to the capacity Size II,III and Size I,IV At the same time, the following reference direction vector is calculated based on the agent target point and agent position information in the sample:

[0335]

[0336] Wherein formula (32-a) represents the target direction in the RT task, and formula (32-b) represents the collision direction in the OAT task. Based on the two environmental information in formula (32), this step gives a collision avoidance direction e obstacle,⊥ , the constraints of each element in this direction are as follows:

[0337] (e obstacle，⊥ ) T e obstacle =0, (33-a)

[0338] ||e obstacle，⊥ ||e target ||, (33-b)

[0339]

[0340] For states in quadrants I and IV t,I,IV, their final auxiliary directions are as follows:

[0341]

[0342] The two auxiliary directions are applied to the policy network (25) using the following update rules:

[0343]

[0344] Here, arccos(·,·) represents the angular overlap between the action and the auxiliary direction obtained by π. The hyperparameter α a The larger the value, the less the policy network depends on the auxiliary direction. a This prevents the control law from being overly dependent on auxiliary directions in the environment, allowing the strategy network to dominate the generation of the control law.

[0345] The above-mentioned embodiments utilize environmental information to construct auxiliary direction rules, as well as the objective function form and constraint form of the auxiliary direction for updating the policy network, which can guide the intelligent agent to reduce behaviors that violate constraints and improve the training efficiency of the policy network; improve task execution efficiency and system stability, and ensure that the intelligent agent optimizes the training process while efficiently completing tasks.

[0346] In a preferred embodiment of the present invention, step S5 is implemented to construct a conflict detection and task replanning mechanism for multi-objective tasks according to the degree of compensation action, including:

[0347] S5.1 Constructing multi-target STL task specifications:

[0348] The tasks assigned to each agent i include: (1) visiting CT in the time domain [0, T] i public target area, and stay in each area for τ seconds; (2) visit PT in the time domain [0,T] i private target areas, and stay in each area for τ seconds; (3) keep a safe distance s from obstacles obs ; (4) Maintain a safe distance d from other agents col The above multi-objective temporal logic task can be expressed as:

[0349]

[0350] S5.2, controller learning layer optimization:

[0351] Considering that in the CTDE training framework, the value function is the cumulative reward of all agents When the target point of an agent i is switched, its cumulative reward will also change, that is,

[0352]

[0353] Among them, k i represents the target point of agent i at the current moment. At time t, the task k of agent i i Completed and switched to k i ′≠k i,t , at this time, due to task k i ′ has not yet been completed, so the agent reward at time t+1 is Make That is, switching the target leads to a decrease in the cumulative estimated reward. Therefore, this step requires embedding the value network Q with a unique hot vector e i ,i∈{1,2,…,N}, to represent the current The set of priorities of all agents evaluated at the current moment

[0354] rank t =(rank t，i |i∈{1,2,…,N}), (38)

[0355] Among them, the execution order of all subtasks of agent i is from index 1 to N i rank t,i Determine, for example, rank t,i =(1,2,3) means that the order of subtask execution of agent i is 1,2,3. And for each agent’s strategy network π i (s t ), just embed your own one-hot vector e i,t , that is, the control law of agent i satisfies at every moment At the same time, in order to reduce the curse of dimensionality of the one-hot vector caused by the large number of target points, this step assigns agent i at time t to task k. i The unique hot vector of is represented using the following fixed-length form:

[0356]

[0357] Among them, splitFactor is a positive integer hyperparameter, which represents the discrimination of the task encoding vector. By encoding the task processed by the agent at the current time t Add to the experience replay pool to form a state-action-reward-hot vector experience replay pool:

[0358]

[0359] The final value function and policy network optimization objectives are as follows:

[0360]

[0361] in, represents the goal k of agent i i The corresponding vector, its policy network Input a vector containing only i θ q ,θ′ q Represent the network parameters of the value function and the target value function respectively, Represents the policy network π i When agent i reaches target point k at time t i to k i ′, the collection rule of robust value reward is defined as follows:

[0362]

[0363] When task k i Completed, its subsequent cumulative robust reward value will remain At this time, task k i The cumulative robust value of is not easy to change to below k i ′, so that k i Priorities are more stable.

[0364] S5.3, construct a priority network:

[0365] Each agent i reads each subtask The cumulative robust value vector from the controller learning layer:

[0366]

[0367] in, Indicates that agent i has The cumulative robustness value of each task in the time domain [0, T]. For each multi-objective task of agent i that can be represented by the STL specification (36), its priority network is expressed as a probability output network as follows:

[0368]

[0369] Among them, N i =GT i +PT i represents the sum of public and private access tasks assigned to agent i in the task time domain [0, T]. Before each round of training, the experience pool of all task robust trajectories of each agent i collected in the previous training rounds is replayed Extract an experience from it and calculate each target point k i Cumulative Robust Vector And input the vector of formula (43) into the network (44) to obtain the priority probability Determine the task priority rank based on the order of probability from large to small i , and then input into the controller learning layer to guide the controller according to the rank i Switch and train the encoding vector.

[0370] S5.4, Robust trajectory data processing:

[0371] Step (a): Extract time nodes to detect collision. In the time domain T, first extract the detection time point t check , denoted as t c , and a small time slice [T start ,T end ], the fragment must meet certain requirements,

[0372] t c ∈[T start , T end ], (45-a)

[0373] |T start -T end |≥τ (45--b)

[0374] T end ≤T, (45-c)

[0375] Subtask The intercepted batch robust trajectory is expressed as follows:

[0376]

[0377] Where S = Size ρ Indicates the batch size of the extracted fragments. For each row of sample data Calculate whether it has suffered a conflict and detect whether the exploration efficiency is low, that is, whether the average rising rate of the accumulated robust value in the slice meets the requirements. The slice robust change rate is as follows:

[0378]

[0379] Among them, speed is the maximum limit of the agent's control law.

[0380] Step (b): Detect the size of the conflict at the sampling time point. This step designs a robust baseline for each agent to meet different robust value change requirements. Multiple To perform different robust value filtering on different subtasks, in order to reduce the probability of individual subtasks being judged as conflicts due to their robust values ​​being lower than the baseline values, the following baseline cluster is constructed:

[0381]

[0382] According to the task k of agent i at the current moment i , to increase the sampling sample accordingly, k i Baseline The batch size of the samples used to prevent baselines for other subtasks and task k i ′’s robust trajectory is overfitted to reduce the misjudgment of conflicts.

[0383] Step (c): Re-plan the priority based on the conflict value. Calculate the performance of each agent i in its respective task k at random time k. i Compensation action The scaling factor of the task is updated according to the size of the compensation action, and then the slice robust trajectory (46) is scaled accordingly.

[0384]

[0385] Among them, the scaling factor Measure the task k performed by agent i i The number and size of the compensation action. The scaling rules (49-c) and (49-e) indicate that only the task k that is in progress at time k i The robust value of the fragment is scaled, and the robust values ​​of the remaining subtasks remain unchanged. At the same time, the robust values ​​of the remaining agents in conflict are scaled as shown in formula (49-f), where represents the set of agents that visit the same area as i and also suffer from conflicts. At this time, in a sampling sample, for all N agents i i The cumulative robust value of a subtask in the time domain [0, T] is calculated as follows:

[0386]

[0387] And the priority network parameters of each agent i are updated using the following cumulative robust weighted cross entropy objective function:

[0388]

[0389] in, It means that the robust trajectory of each task of agent i is sampled in Monte Carlo in the experience replay pool, and the average is taken to obtain the target loss function value after calculating the value.

[0390] In a preferred embodiment of the present invention, step S6 is implemented to iteratively train controller parameters to provide a control law online to meet the STL task requirements, including:

[0391] S6.1, Initialization: Initialize all network parameters, agent initial position, multi-intelligence multi-objective STL task form, and network hyperparameter settings.

[0392] S6.2, collect and update the experience revisit pool: take each execution of the policy network T times as a training cycle, collect the position information, action information, rewards, and task encodings of T agents as a group. When the amount of data exceeds the capacity of the experience revisit pool, overwrite the data at the corresponding time in the experience revisit pool from the beginning.

[0393] S6.3, continuously switch task encoding in each training cycle: according to the priority of each agent in the current cycle, when task k i After completion, switch to the next task k i ′, and at the same time the encoding vector Update the input of the value network and the policy network, and use the gradient descent method to update all networks.

[0394] S6.4, continuously update the priority: After each cycle is executed, the task priority of each agent in the next cycle is determined according to the probability distribution of the output after the cumulative robust value is input to the priority network. At the same time, the priority network parameters are updated. Until in P cycles, the position trajectory of all agents from time 0 to time T in each cycle meets the STL task specification, or the MAX is terminated. epoch After cycles of training, the training is terminated and the policy network parameters of the final N agents are obtained.

[0395] S6.5, test the performance of the policy network: This step does not update the policy network. T In a test cycle, the initial position is selected for each agent, and the control law is continuously output according to the position information from time 0 to T, and applied to the agent dynamic system to obtain the agent position trajectory of length T. The position trajectory of each agent is tested to determine whether it meets the STL specification and P T The number of P tests that meet the requirements success .

[0396] The multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks provided by the embodiment of the present invention can enable the multi-agent system to meet the multi-objective STL task requirements while meeting the collision avoidance and obstacle avoidance requirements under complex temporal logic constraints.

[0397] In some specific embodiments, the initial value distribution of all network parameters follows a normal distribution. α=25,β=-70,γ=-50,maximum training period T epoch =5500, learning rate lr a =0.001,lr c =0.002, Experience replay pool capacity Size ρ =1200, the size of the batch of data extracted when the policy network is updated π =256, robust trajectory extraction batch size Size ρ =128, robust baseline parameter β b =1000, robust baseline threshold ρ max =0.25, auxiliary direction objective function weight α a =20, reward discount factor λ=0.9, network parameter soft update factor α q =0.01, scaling factor update step length γ d =100, the optimizer used to update the network parameters is Adam.

[0398] The following further describes the technical solution provided by the embodiment of the present invention in conjunction with the accompanying drawings and through simulation examples. Figure 2 , the scale of the multi-agent system is N = 2. Assume that the initial position centers of the unmanned vehicle are [0.10], [10,20], and the length and width of the initial position are [2,2], respectively represented by the yellow area. The scene includes an obstacle with a center of [10,10] and a radius of 1.5, and an agent feasible area of ​​size [22,22], respectively represented by a gray circular area and a black frame line. The four time-limited arrival task target points are represented by green dotted areas, with center positions of [5,5], [15.5], [15,15], and [5,15], and the length and width of the area are [2,2]. Each agent i∈{1,2} has two public access areas And a visit mission area of ​​its own At the same time, the agents maintain a certain safe distance d safe =4.

[0399] Normalize the above sequential logic tasks:

[0400]

[0401] This complex sequential logic task indicates that agents 1 and 2 need to reach their respective target areas within the time interval [0,30] and stay in the target area for 3 seconds. Indicates that Agent 1 needs to reach the target point [5, 15] within the time constraint, and the length and width of the square area are 2; when multiple agents complete the multi-target time-limited arrival task, they must not touch each other at all times. Figure 1Obstacles and walls in the system. For example, the temporal logic formula This means that the obstacle with the center at [10,10] and the radius of 1.5 must not be touched within the total STL task time [0,30]. This means that the center distance between Agent 1 and Agent 2 cannot be less than 4.

[0402] In this embodiment, in an offline state, according to a given multi-objective STL task, the policy network, value network, reward function, robust filtering baseline, self-attention network, auxiliary direction, priority network and conflict detection mechanism parameters are initialized for each agent. For each agent, a square area of ​​the initial position is given. Please refer to Figure 2 .

[0403] Each agent randomly selects an initial position, and in one cycle, continuously collects corresponding information from the experience replay pool. At each moment, each network parameter is updated based on a given policy network and compensation mechanism. During this period, the agent's task encoding vector is continuously updated according to the task completion status. After every 30 time steps of training, the robust filtering baseline is updated again, and the task priority and priority network used in the next cycle are updated according to the conflict detection mechanism. It is iterated until the STL tasks of all agents are satisfied in 100 cycles. The present invention compares the training results with and without the conflict detection mechanism, and uses the same training parameters and process to finally obtain the policy network parameters and task priorities of each agent, and obtain the motion trajectory diagram of each agent within 30 time steps. Please refer to Figure 3 and Figure 4 .

[0404] The goal of agent A Recorded as task (0,1,2), similarly, the goal of agent B is Also written as (0,1,2).

[0405] For the motion trajectory of the agent without adding the detection mechanism, please refer to Figure 3 This figure does not include priority replanning, but only uses the priority network and the native cumulative robust value. Intuitively, because the agents visit the same target point 0, there will be a collision avoidance conflict, causing agent B to wait, thus failing to meet the task requirements.

[0406] For the motion trajectory diagram of the intelligent agent with the detection mechanism added, please refer to Figure 4 Agent B chooses the access sequence (1, 2, 0) that has the largest cumulative robustness value and meets the requirements of temporal logic, while Agent A re-plans its task priority to (2, 1, 0) to reduce the conflict with Agent A, thus completing the global MAMO-STL task.

[0407] During the training process, the cumulative robustness value change curve of each agent corresponding to each target point in each training cycle (30 control moments) is shown in Figure 5 , Figure 6 , Figure 7 and Figure 8 .

[0408] For the original cumulative robust value change curve of each target point without the robust rate conflict detection mechanism, please refer to Figure 5 and Figure 6 , where even if agents A and B conflict at task 0, the relative order of their cumulative robust values ​​remains almost unchanged, so in the end both agents will not change their task priorities and will follow this priority to train the controller parameters, ultimately violating the task requirements.

[0409] For the cumulative robust value change curve with the robust rate conflict detection mechanism of the present invention, please refer to Figure 7 and Figure 8 . When Agent A was trained for about 2300 rounds, the relative size of the robustness value between Task 1 and 0 changed, while the relative size of the accumulated robustness value between Task 2 and 0 of Agent B changed around 2600 rounds, that is, when Agent B finally judged that its execution priority was (0,2,1), it was able to explore the path with the least conflict and that met the requirements of temporal logic. So far, the priority of each agent has also changed accordingly, with Agent A being (2,1,0) and Agent B being (0,2,1).

[0410] Priorities are re-planned because during training, agents visit the same target point and conflict occurs. At this time, the scaling factors of their respective tasks The conflict factors of each task (0, 1, 2) of each agent are accumulated or weakened, and finally the conflict factors corresponding to each task (0, 1, 2) of each agent are (4.997, 2.153, 1) and (0.212, 0.416, 1) respectively. Among them, the conflict factor of task 0 of agent A increases cumulatively to 4.997, indicating that agent A is constantly experiencing collision avoidance conflicts when interacting with B at task 0, and therefore the scaling factor value is constantly accumulated, while the scaling factor of agent A at task 1 is 2.513, which indicates that the number or size of its conflicts is less than the conflict at task 0. Based on the changes in the above factors, since the cumulative robustness value of each subtask is itself a negative number, after multiplying by the scaling factor, its cumulative robustness value will decrease, thereby changing the relative size of the cumulative robustness values ​​between tasks and affecting the priority arrangement. For example, the priority of task 1 of agent A gradually exceeds the priority of task 0, and finally each agent meets the requirements of the time-limited public access task and private access task.

[0411] The above embodiment of the present invention is aimed at the problem of task planning and control of multi-agent system based on reinforcement learning under multi-objective temporal logic. Considering that the multi-agent system lacks a collaborative ability to perceive the degree of completion of each STL task when interacting with the environment, a self-attention mechanism is introduced to extract the degree of conflict impact between agents and generate compensation actions to correct the actions output by the controller. It also involves the reuse of robust value information generated during the training process, and it is necessary to extract the robust value fragments that violate the task robustness requirements and use them to improve the training efficiency of the self-attention module; it involves the re-planning of the priority order of multi-objective STL tasks during the training process, and it is necessary to continuously record the number of conflicts between agents during the training process and continuously update the priority so that the trained controller meets the STL requirements.

[0412] The above are only preferred embodiments of the invention and are not intended to limit the scope of the invention. That is, any equivalent changes and modifications made according to the content of the patent application scope of the present invention should be within the technical scope of the present invention.

Claims

1. A multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks, characterized in that: include: S1, establish the reinforcement learning controller training framework for signal timing logic tasks in multi-agent systems; S2, establish a self-attention action compensation mechanism among multi-agent systems; S3, constructing a robust value filtering baseline corresponding to the signal timing logic task according to the controller training architecture established in step S1; S4, generates soft constraints on the agent’s actions based on environmental information; S5, constructs the conflict detection and task replanning mechanism of multi-objective tasks according to the degree of compensation action; S6, based on the constraints set in steps S2, S3, S4, and S5, iteratively train the controller parameters established in step S1 to provide an online control law that meets the requirements of the multi-objective sequential logic task.

2. According to the method for planning and controlling multi-agent reinforcement learning tasks driven by spatiotemporal tasks in claim 1, it is characterized in that: S1, establish the reinforcement learning controller training architecture for signal timing logic tasks in multi-agent systems, including: S1.1, establish a multi-agent state space model: The multi-agent system consists of n agents i, The multi-agent state space model is described as: in, and Represents the horizontal and vertical position information and control input information of agent i respectively; the state and control input of the agent are bounded and satisfy the constraints as well as S1.2, based on the multi-agent state space model, a multi-agent temporal logic task normalization framework is established: Based on STL formal statements, describe the multi-agent signal timing logic task and establish a mathematical model of the task scenario: Where φ′ and φ″ are STL formulas, in the form of equation φ; the time interval [a, b] represents a series of time steps [a, a+1, …, b]; and ∧ are the logical operators "NO" and "AND"; G, F, and U are the temporal operators "global", "final", and "until"; for a discrete-time signal x, Indicates that the signal x at discrete time t satisfies the constraint φ; Indicates that the constraint φ is not satisfied; It means that φ′ and φ″ are satisfied at the same time; Indicates that the constraint φ must always be satisfied within the time interval [a,b]; Indicates that the constraint φ must be satisfied at least at one time in the future time interval [a, b]; operator It means that the constraint φ″ must be satisfied at least once in the time interval [a, b], and the constraint φ′ must be continuously satisfied before the constraint φ″ is satisfied; STL is based on multiple predicate logics μ, each of which consists of a column defined in the system state space. The predicate function h in denoting the actual physical meaning is: Decision; The condition h(x) ≥ 0 is used to mathematically describe that the actual system is in the required set; The predicate v is defined as follows: STL provides a robust semantics that defines a real-valued function ρ φ , used to indicate that if ρ φ (x,t)≥0, then x satisfies φ, and the degree to which the signal x satisfies or violates φ is recorded; The robust semantics is defined as: Among them, the real-valued function ρ φ It is used to describe the degree to which the system trajectory satisfies a specific STL constraint and is expressed as a real value with robust semantics; the positive and negative signs indicate the degree to which the system state trajectory violates or satisfies the constraint; in the mathematical model of the task scenario, when two constraints intersect, the minimum value of the robustness should be taken to ensure that they can be continuously satisfied at the same time; when two constraints are combined, the maximum value of the robustness of the two constraints is taken to ensure that both can be satisfied once; when with the global operator G, the minimum value of the robustness of the two constraints is taken; when with the final operator F, the maximum value of the robustness is taken; when with the until operator U, the time node needs to be introduced At this time point, the system state trajectory satisfies both constraints φ″ and φ′, that is, and When ρ is satisfied φ When (x,t)≥0, it means S1.3, construct multi-agent temporal logic task rules: The signal timing logic task of agent i is: in, are the horizontal and vertical spatial positions of agent i at time t; Indicates that agent i needs to be in the target area r i The task is expressed as a predicate function: S1.4 Establish a multi-agent reinforcement learning MARL framework: Formulate multi-agent signal temporal logic tasks as Markov games Includes several states and actions Each action can transform the current state into a new state and obtain a reward; That is, the set of agents in the Markov game; denote all agents as Where N is the total number of agents; represents the joint state set of all agents; represents the spatial position of agent i at time step t, Represents the joint state of N agents; The joint set of actions of all agents; represents the actions taken by agent i in two directions at time step t, represents the joint action of N agents; State transition probability function; where P(s t+1 =s′|s t =s,a t =a) maps the state s t+1 In status t Take action a t The state distribution after that, P:S×A×S′→[0,1]; Reward function; Indicates that in state s t Take action when The immediate reward generated later; Setting: The goal of each agent i is to learn a parameter θ i Strategy π i (s t θ i ) to maximize their cumulative rewards; the interactions between agents are through their actions This is achieved by considering the joint actions of each agent; these actions will affect the state of the environment s. t And each agent receives a reward signal r i ; Under the signal timing logic STL task, a training framework for multiple agents with centralized training and decentralized execution is established. Consider a game with N agents, whose game space is shaped like a tuple The policy network parameters of the agent are θ=(θ1,…,θ N ), and define the set π={π1,…,π N } is the strategy network of all agents; The expected evaluation function corresponding to the policy network action of agent i is defined as Then the parameter θ i The expected gradient of is written as: in, is to take into account all the action information a of the other agents t The joint value function of agent i, s t is the state information of each agent; Value Function It can measure the STL robustness value of the agent at the current state s; Each They are all learned separately, and the agents have independent reward function structures, including collision avoidance penalties in static environments; The solution to the policy gradient is based on the experience replay pool The form of taking the mean after sampling is: Among them, π i Similar to the policy network in DDPG to give continuous action values; experience replay pool Contains the state-action-reward-new state tuple This tuple is called an experience, and The experience of all agents is recorded; Based on this experience replay pool Centralized action-value function for multiple agents The parameter update rules are as follows: Among them, π={π′1,…,π′ N } is a function with target value parameter θ′ i The target strategy set; S1.5, establish the multi-agent reward function design rules under the STL specification: For each agent, the RT of the STL task reaches the task part F [a,b] G [0,τ] , only considering its robust value within τ time steps; The historical state of τ time steps is expressed as And by inputting s into the policy network t-τ:t And get action a t , and then execute the action into the agent dynamic system to obtain the state s at the new moment t+1 ; At time t+1, the τ-step history state will be updated and the corresponding RT reward R will be generated i (s t-τ:t ,t), which is defined as: For the collision avoidance task and obstacle avoidance task, the reward function is described as: Among them, R i (s t-τ:t ,t) CAT Usually represents the closest distance between the current agent i and other agents, which is greater than d safe It is safe; Based on the components of these reward functions, the reward function of any agent is obtained as: R i (s t-τ:t ,t)=α*R i (t) RT +β*R i (t) OAT +γ*R i (t) CAT ,(12)Wherein, α, β, and γ are the weight hyperparameters of the rewards for each part of the STL task.

3. According to the method of claim 2, the multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks is characterized in that: S2, establish a self-attention action compensation mechanism among multi-agent systems, including: S2.1, preprocessing future time robust trajectory data: Based on the motivation of utilizing future robust information, the data of the experience replay pool is expanded to Contains tuples in, represents the robust value of the time-limited arrival task at time t, represents the robustness value of the collision avoidance task part of the STL task of agent i; use To describe The capacity of the robust value in , where represents the complete robust trajectory from 0 to T The number of,T represents the entire time domain of the STL task; S2.2, construct a robust trajectory self-attention mechanism module: Based on the information sampled by multiple agents at time k A self-attention mechanism based on robust value trajectories is designed to extract the influence relationship between agents. The weight generation network of the self-attention module is: in, represents the impact of the robust information of agent j on agent i; S2.3, construct the self-attention network parameter update rules during training: At each training moment, a compensation time k∈[0,T] is first randomly selected; By making the sum of the cumulative rewards of the trajectory generated after compensation greater than the reward before compensation, the compensation network parameter θ is designed. c ,W K W Q and W V The update rule is: Among them, V c Is the agent accepting compensation action The maximum cumulative reward value that can be achieved after , is recorded as the compensation value function, is the state and action data of the same sample in the robust data extraction process, It means that only the compensation action is performed on agent i in all action spaces. Agent joint action, expressed as Compared with Only the i-th element is replaced; Add constraints to the update rule: Among them, V is the original value function, and the trainable parameter is θ q ; The constraints represent the experience pool and The sampled data has the same index position in the experience pool, that is, the state-action-reward experience and robust value experience generated at the same time in different rounds of the training process; At the same time, update π according to the gradient ascent method i Parameters: The parameters of both value functions are updated using the objective function, and the update rules are as follows: Among them, V c ′ and V′ are the two objective value functions of the compensation value function and the original value function, and their parameters are θ′ q,c and θ′ q ; Update these parameters according to the learning rate lr and the target value function update factor α:

4. The method for planning and controlling multi-agent reinforcement learning tasks driven by spatiotemporal tasks according to claim 3 is characterized in that: In step S2.1, use To describe The capacity of the medium robust value is: Randomly select a timestamp k and calculate the batch size according to the batch size. ρ Extract multiple robust value sequences to obtain a data set as the input of Shape; the data set is: in, The data structure is that for the part of agent i from time k to T, S = Size ρ Strip robust trajectory; Sample multiple collision avoidance information trajectories: Among them, Size Col =Size ρ Indicates the number of samples extracted; Combine each element of these two inputs into Get robust information input:

5. The method for planning and controlling multi-agent reinforcement learning tasks driven by spatiotemporal tasks according to claim 4 is characterized in that: The self-attention model in S2.2 implements an approximate representation of the influence weight, specifically: The query vector of agent i at time step t is expressed as in, Size ρ A random sample [:] index means extracting the corresponding fragment information of all agents, W Q is the trainable parameter matrix in the model; The key vector and value vector information of agent i and agent j at time step t are expressed as and The calculation method of the self-attention influence weight between agents is: Among them, d represents the dimension of parameter matrices Q and K, and weights is the influence factor between agent i and agent j at time t; Based on the impact factor in agent j In the case of , the action compensation coefficient that agent i should make for: Among them, ⊙ is the bitwise product of the matrix; the physical meaning of the compensation factor is that if the input of j The future robust value after the k moment represented by must meet the requirements of the STL specification. At this time, agent i should additionally execute the original strategy π i Some compensatory actions other than For agent i, in the combination of different agents The compensation factor Afterwards, the final compensation factor at sampling time k is given as: in, represents the compensation response that agent i should make at time k; Normalize the final compensation factor: Among them, MLP is a fully connected layer neural network. is the compensation action network MLP of agent i i At the same time, the activation function tanh is used to limit the size of the compensation action to prevent the gradient explosion during training. Applying compensation actions directly to the agent’s original policy network The synthesized action is expressed as:

6. The method for multi-agent reinforcement learning task planning and control driven by spatiotemporal tasks according to claim 1, characterized in that: S3, constructs the robust value filtering baseline corresponding to the STL task according to the task framework, including: S3.1, Design of robust value filtering baseline: Subtract the actual robust value from the baseline to obtain the filtered robust value information: Among them, B(k) represents the baseline value of the robust filter at time k, that is, the expected robust value of the [0,T] time domain series at that time, and k:T represents the interception of records between time k and T of each complete record; S3.2, design rules for filtering robust information: If the robust trajectory is evenly distributed around the baseline and above the baseline, we get The compensation action network will not be activated and no compensation action will occur.

7. The method for planning and controlling multi-agent reinforcement learning tasks driven by spatiotemporal tasks according to claim 6 is characterized in that: The form and parameters of the robust value filtering baseline in S3 are restricted so that the robust data after baseline filtering can fully express the satisfaction of the task at each control moment. Specifically: r max ≥Baseline(T)≥0,(30-c) Update the baseline parameters by gradient ascent: When the parameters of the Baseline converge and the robust trajectory is above the Baseline, at time T, the robust value is greater than 0, indicating that the signal timing logic task of agent i is satisfied.

8. The method for multi-agent reinforcement learning task planning and control driven by spatiotemporal tasks according to claim 1, characterized in that: S4, generates soft constraints on the agent's actions based on environmental information, including: S4.1, construct the positive direction of the auxiliary direction coordinate system: Revisiting the pool based on experience Extracted from t information, and obs , will s obs As the origin in space, As the positive direction of the x-axis in this space, e x ; S4.2, generate auxiliary directions: In setting the positive direction e x After that, the Size sampled from D train The tuple is divided into two parts: s t,II,III and t,I,IV , representing the second and third quadrants and the first and fourth quadrants of the sampled data respectively; these two parts of data correspond to the capacity Size II,III and Size I,IV ; S4.3 calculates the following reference direction vector in the sample based on the agent target point and agent position information: Wherein formula (32-a) represents the target direction in the RT task, and formula (32-b) represents the collision direction in the OAT task; S4.4, determine a collision-avoiding direction e based on the two environmental information of the reference direction vector obstacle,⊥ , the constraints of each element in this direction are: For states in quadrants I and IV t,I,lV , their final auxiliary directions are as follows: And the two auxiliary directions are applied to the policy network update using the following update rules: where arccos(·,·) represents the angular overlap between the action and the auxiliary direction obtained by π; the hyperparameter α a The larger it is, the less dependent the policy network is on the auxiliary directions.

9. According to claim 1, a multi-agent reinforcement learning task planning and control method driven by spatiotemporal tasks is characterized in that: S5, constructs a conflict detection and task replanning mechanism for multi-objective tasks based on the degree of compensation action, including: S5.1, Constructing multi-objective STL task specifications: The tasks assigned to each agent i include: visiting CT in the time domain [0,T] i public target area, and stay in each area for τ seconds; visit PT in the time domain [0,T] i private target areas, and stay for τ seconds respectively; keep a safe distance d from obstacles obs ; Keep a safe distance d from other agents col ; The multi-objective STL task is expressed as: S5.2, controller learning layer optimization: Under the training framework, the value function is the cumulative reward of all agents When the target point of an agent i is switched, its cumulative reward will also change, that is, Among them, k i represents the target point of agent i at the current moment. When the task k of agent i at time t is i Completed and switched to k i ′≠k i,t , at this time, due to task k i ′ has not yet been completed, so the agent reward at time t+1 is Make That is, switching targets leads to a decrease in the cumulative estimated reward; Embed the value network Q with a one-hot vector e i , i∈{1,2,…,N}, to represent the priority set of all agents at the current moment evaluated by the current Q rank t =(rank t,i |i∈{1,2,…,N}), (38) Among them, the execution order of all subtasks of agent i is from index 1 to N i rank t,i Sure, Assign agent i to task k at time t i The one-hot vector of is represented using a fixed-length form: Among them, splitFactor is a positive integer hyperparameter, which represents the discrimination of the task encoding vector; Encode the task that the agent is processing at the current time t Add to the experience replay pool to form a state-action-reward-hot vector experience replay pool: Get the final value function and policy network optimization target: in, represents the goal k of agent i i The corresponding vector, its policy network Input a vector containing only i θ q ,θ′ q Represent the network parameters of the value function and the target value function respectively, Represents the policy network π i Network parameters; When agent i reaches target point k at time t i to k i ′, the collection rule of robust value reward is defined as follows: When task k i Completed, its subsequent cumulative robust reward value will remain At this time, task k i The cumulative robust value of is not easy to change to below k i ′, so that k i Priorities are more stable; S5.3, construct a priority network: Each agent i reads each subtask inde=1,2,…,N i The cumulative robust value vector from the controller learning layer: in, Indicates that agent i has The cumulative robustness value of a task in the time domain [0,T]; For each agent i’s multi-objective task that can be characterized by the STL specification, its priority network is expressed as a probability output network as follows: Among them, N i =GT i +PT i represents the sum of public and private access tasks assigned to agent i in the task time domain [0, T]; before each round of training, the experience pool of all task robust trajectories of each agent i collected in the previous training rounds is replayed Extract an experience from it and calculate each target point k i Cumulative Robust Vector k i =1,2,…,N i , and input the accumulated robust value vector into the probability output network to obtain the priority probability Determine the task priority rank based on the order of probability from large to small i , and then input into the controller learning layer to guide the controller according to the rank i Switch and train the encoding vector; S5.4, Robust trajectory data processing: S5.4.1, extract time nodes to detect collision: In the time domain T, first extract the detection time point t check , denoted as t c , and a small time slice [T start ,T end ], the fragment must meet certain requirements, t c ∈[T start ,T end ], (45-a) |T start -T end |≥τ, (45-b) T end ≤T, (45-c) for subtasks The intercepted batch robust trajectory is expressed as follows: Where S = Size ρ Indicates the batch size of the extracted fragments; for each row of data in the sample Calculate whether it has suffered a conflict and detect whether the exploration efficiency is low, that is, whether the average rising rate of the accumulated robust value in the slice meets the requirements; the slice robust change rate is as follows: Among them, speed is the maximum limit of the agent's control law; S5.4.2, detect the amount of conflict at the sampling time point: A robust baseline is designed for each agent to meet different robustness value change requirements; multiple To perform different robust value filtering on different subtasks, in order to reduce the probability of individual subtasks being judged as conflicts due to their robust values ​​being lower than the baseline values, the following baseline cluster is constructed: According to the task k of agent i at the current moment i , to increase the sampling sample accordingly, k i Baseline The batch size of the samples used to prevent baselines for other subtasks and task k i The robust trajectory of i is overfitted to reduce the misjudgment of conflicts; S5.4.3, Re-prioritization based on conflicting values: Calculate the performance of each agent i in their respective task k at random time k i Compensation action The scaling factor of the task is updated according to the size of the compensation action, and then the slice robust trajectory (46) is scaled accordingly; Among them, the scaling factor Measure the task k performed by agent i i The number and size of the compensation action. The scaling rules (49-c) and (49-e) indicate that only the task k that is in progress at time k i The robust value of the fragment is scaled, and the robust values ​​of the remaining subtasks remain unchanged; at the same time, the robust values ​​of the remaining agents in conflict are scaled as shown in formula (49-f), where represents the set of agents that visit the same area as i and also suffer from conflicts; at this time, in a sampling sample, for all N agents i i The cumulative robust value of a subtask in the time domain [0, T] is calculated as follows: And the priority network parameters of each agent i are updated using the following cumulative robust weighted cross entropy objective function: in, It means that the robust trajectory of each task of agent i is sampled in Monte Carlo in the experience replay pool, and the average is taken to obtain the target loss function value after calculating the value.

10. The method for multi-agent reinforcement learning task planning and control driven by spatiotemporal tasks according to claim 1, characterized in that: S6, iteratively train the controller parameters to give the control law online to meet the STL task requirements, including: S6.1, Initialization: Initialize all network parameters, initial position of intelligent agents, multi-intelligence multi-objective STL task form, and hyperparameter settings in the network; S6.2, collect and update the experience revisit pool: take each execution of the policy network T times as a training cycle, collect the position information, action information, rewards, and task encoding of T agents as a group, and when the amount of data exceeds the capacity of the experience revisit pool, overwrite the data at the corresponding time in the experience revisit pool from the beginning; S6.3, continuously switch task encoding in each training cycle: according to the priority of each agent in the current cycle, when task k i After completion, switch to the next task k i ′, and at the same time the encoding vector Update the input of the value network and the policy network, and use the gradient descent method to update all networks; S6.3, continuously update the priority: each cycle is executed, the task priority of each agent in the next cycle is determined according to the probability distribution of the output after the cumulative robust value is input to the priority network; at the same time, the priority network parameters are updated; until in P cycles, the position trajectory of all agents from time 0 to time T in each cycle meets the STL task specification, or the MAX is terminated. epoch After training for cycles, the training is terminated and the policy network parameters of the final N agents are obtained; S6.4, Test Strategy Network Performance: In P T In a test cycle, an initial position is selected for each agent, and the control law is continuously output according to the position information from time 0 to T, which is applied to the agent dynamic system to obtain the agent position trajectory of length T; and the position trajectory of each agent is tested to determine whether it meets the STL specification and P T The number of P tests that meet the requirements success .

Images