Strategy generation and decomposition method based on linear temporal logic control

Abstract

The invention relates to the technical field of agents, in particular to a strategy generation and decomposition method based on linear temporal logic control, which comprises the following steps of: standardizing a plurality of agent system behaviors by using a GR (1), synthesizing and formalizing a strategy into accessibility, calculating a fixed point set to determine a state meeting an agent target, and constructing an automaton to realize the GR (1) specification; decomposing the synthetic strategy automaton into an individual strategy automaton, and enabling each agent to act based on local observation and decision of the agent; constructing a decentralized experience replay Q learning algorithm with synchronization and decomposition strategies; a DER-QL algorithm synchronous automaton is used for buffering a strategy learned from a high-level synthesis strategy, so that learning of a low-level MARL is guided. According to the method, the problem of performance reduction caused by lack of coordination when behaviors between agents are mutually influenced by an existing DQL algorithm is solved; and unreasonable decisions are easy to repeatedly make, so that the learning efficiency is low and the convergence speed is slow.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agent technology, and in particular to a strategy generation and decomposition method based on linear temporal logic control. Background Art

[0002] A multi-agent system is a distributed computing framework involving multiple agents that interact cooperatively or adversarially in a shared environment. Multi-agent reinforcement learning has potential, but significant challenges arise in the presence of non-Markov tasks, where future states depend on past states other than the current state. When the Markov decision assumption fails, the performance of multi-agent reinforcement learning will significantly degrade. When dealing with multi-task specifications, problems such as slow convergence and sparse rewards will also arise.

[0003] Linear Temporal Logic (LTL) is a well-established formal method that specifies desired system properties such as safety and liveness. When applied to multi-agent systems (MAS), it can effectively capture temporal properties including safety, liveness, and reactive interactions; combining LTL with multi-agent reinforcement learning (MARL) is a very active research area.

[0004] The paper Decentralized Q-Learning in Zero-sum Markov Games is a decentralized learning algorithm DQL, in which each agent learns independently and updates its strategy only based on its own local observations and rewards. In a multi-agent system, the behaviors of agents influence each other, and lack of coordination may lead to a decline in overall performance. In addition, the DQL algorithm does not adopt an experience replay mechanism, which means that it can only rely on current real-time interaction experience during the learning process, while ignoring valuable information from the past. In a complex environment, agents may encounter similar situations many times, but because they cannot review and learn from past experiences, they may repeatedly make suboptimal decisions, resulting in low learning efficiency and slow convergence. Summary of the invention

[0005] To address the shortcomings of existing methods, this paper uses a new synchronous decentralized training paradigm to provide multiple agents with strategies synthesized by GR(1); formalizes the strategy synthesis problem as a reachability problem of states that satisfy system goals; decomposes the comprehensive strategy into independent strategies, and uses the DER-QL algorithm to guide distributed learning in MARL.

[0006] The technical solution adopted by the present invention is: a strategy generation and decomposition method based on linear temporal logic control comprises the following steps:

[0007] Step 1: Use GR(1) to regulate the behavior of several agent systems, formalize the strategy synthesis into reachability, calculate the fixed point set to determine the state that satisfies the agent's goal, and construct an automaton to implement the GR(1) specification so that the agent satisfies the safety and liveness properties under the assumptions of the environment;

[0008] As a preferred embodiment of the present invention, step one specifically includes:

[0009] Step 11, construct GR(1) specification constraints;

[0010] Step 12: Use fixed point calculation to reversely iterate the target state to calculate the set of predecessor states to reach the target state, and construct a synthetic strategy automaton Where Q is a finite state set, ∑ is the action alphabet, δ is the transition function, q0 is the initial state, and F is the set of accepting states;

[0011] As a preferred embodiment of the present invention, the fixed point calculation includes:

[0012] Initialization satisfies the activity properties of the system The state collection

[0013] Initializes liveness properties that do not satisfy the environment The state set

[0014] For each state index i, j, calculate Represents the entire state space; is a Boolean function, indicating that the state s satisfies the invariant of the system Indicates that there is a successor state s′ that belongs to the state set G s [i][j];

[0015] calculate

[0016] Update j = j + 1 and perform convergence judgment until G s [i][j+1]=G s [i][j] and G e [i][j+1]=G e [i][j], the fixed point is reached and the iteration ends.

[0017] Step 13: Based on the fixed point set G s and G e And the invariant and Building a synthetic strategy automaton

[0018] As a preferred embodiment of the present invention, step 13 specifically includes:

[0019] Initialize the state set Q, the transition function δ and the acceptance state set F, and add the initialization state q0 to the pending set next;

[0020] Take a state s from next and determine the index of s (i, j); if j = 0, get the next system goal to be verified and update the state index j;

[0021] For each invariant and The potential successor state s′ of i is checked to see whether s′ belongs to the specified set G of i in j. s or G e In the set; if so, calculate the transfer action a and record the transfer δ(s,a)←s′; if s′ is already in Q, reuse it; otherwise add s′ to next;

[0022] If no successor state s′ is found when j≠0, terminate; return to the synthetic strategy automaton

[0023] Step 2: Decompose the composite strategy automaton into individual strategy automatons, so that each agent takes actions based on its own local observations and decisions;

[0024] As a preferred embodiment of the present invention, step 2 specifically includes:

[0025] Constructing individual policy automata and parallel combinatorial automata Among them, Q || =Q1×Q2×…×Q N is the Cartesian product of the states of the factorized automaton, is the union of the decomposition automaton alphabets, δ || :Q || ×∑ || →Q || is the transfer function, defined as: δ || ((q1,q2,…,q N ),σ)=(δ1(q1,σ),δ2(q2,σ),…,δ N (q N ,σ)), when Time δ i (q i ,σ)=q i ;q 0,|| =(q 0,1 ,q 0,2 ,…,q 0,N) is the initial state, F || =F1×F2×…×F N is the Cartesian product of accepting states.

[0026] Step 3: Construct a decentralized experience replay Q-learning algorithm with synchronization and decomposition strategies;

[0027] As a preferred embodiment of the present invention, step three specifically includes:

[0028] Step 31: Construct the extended Dec-POMDP model. N represents the set of agents in the environment; S represents the set of environmental states, s0 is the initial state; (O i ,A i ,R i ) is the observation-action-reward triplet of agent i; T is the state transition probability distribution; R is the global reward function; γ is the discount factor; is the individual strategy automaton; is the action labeling function; is the state labeling function; Ψ i is the potential function on each state of the automaton;

[0029] Step 32: For each state index i, The potential value of state q is Ψ i (q) is initialized to 0; enter the iteration loop and set the variable Δ that tracks the maximum change in the potential value in each iteration to 0;

[0030] Record the potential value of the current state q as Ψ old , updates the potential value Ψ of the current state q by maximizing the expected reward plus the discounted future potential value of all possible local actions σ i (q);

[0031] Calculate the update amount of the potential value of the current state |Ψ i (q)-Ψ old |, and compare it with Δ. If the update amount is greater than Δ, then Δ←max(Δ,|Ψ i (q)-Ψ old |);

[0032] After each iteration, check whether Δ is less than the preset threshold ∈ v ; if Δ<∈ v , the iteration stops; otherwise, return to the step of iteratively updating the potential value to continue the next iteration.

[0033] Step 4: Use the DER-QL algorithm to synchronize the automaton buffering strategies learned from the high-level synthesis strategy, thereby guiding the learning of low-level MARL;

[0034] As a preferred embodiment of the present invention, the construction of the DER-QL algorithm includes:

[0035] Initialize the Q value of each agent, define the initial state s of the agent and the state q of the automaton i ;

[0036] The agent observes the local environment state o i ; Select action a from the ε-greedy strategy in the Q table i ; If there are overlapping local actions between any pair of agents, synchronization is performed on the agents’ policies;

[0037] Update the automaton state q i and potential function Ψ i , and calculate the reward R i , update the Q table using the Q learning update rule;

[0038] Update the state s, observation value o and state q for the next iteration;

[0039] Execute the selected action set a and observe the next state s′.

[0040] As a preferred embodiment of the present invention, a strategy generation and decomposition system based on linear temporal logic control includes: a memory for storing instructions executable by a processor; a processor for executing instructions to implement a strategy generation and decomposition method based on linear temporal logic control.

[0041] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, the strategy generation and decomposition method based on linear temporal logic control is implemented.

[0042] Beneficial effects of the present invention:

[0043] 1. Based on the synchronous decentralized training paradigm of GR(1) specification, the strategy synthesis problem is transformed into a reachability problem, and solved by fixed point calculation, the synthesis strategy is decomposed into individual strategies to guide distributed learning; it solves the challenge of adapting to complex dependencies in multi-agent reinforcement learning, especially when dealing with non-Markov tasks, and improves the adaptability and scalability of the system, while traditional methods are often limited in performance when facing such tasks;

[0044] 2. Through the synthesis strategy algorithm, it is ensured that the constructed automaton satisfies the specified invariants and goals, and the entire GR(1) synthesis process can be solved in polynomial time in terms of the state space size and the number of target attributes;

[0045] 3. Construct a DER-QL algorithm with a synchronous reward structure, which can effectively use high-level synthetic strategies to guide low-level learning; through experience replay technology, it improves learning stability and reduces the volatility of intelligent agent decisions, overcomes the shortcomings of traditional distributed Q-learning in coordination and communication, and can use past conversion data for learning multiple times, thereby converging to a better strategy faster and improving the overall learning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the strategy generation and decomposition method based on linear temporal logic control of the present invention;

[0047] Figure 2 It is a comparison chart of the training effects of the method of the present invention and the existing method. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0049] like Figure 1 As shown, a strategy generation and decomposition method based on linear temporal logic control includes the following steps:

[0050] Step 1: Use GR(1) (Generalized Reactivity of rank 1) to regulate the behavior of the multi-agent system, formalize the strategy synthesis as a reachability problem, determine the state that satisfies the agent's goals by calculating the fixed point set, and construct an automaton to implement the GR(1) specification to ensure that the agent meets the safety and activity properties under the assumptions of the environment. The synthesis process can be completed in polynomial time.

[0051] Linear temporal logic (LTL) is used to define the properties of discrete systems. First, the LTL strategy converts the LTL formula into a non-deterministic Buchi automaton, which represents all possible event sequences that satisfy the formula.

[0052] Secondly, by calculating the winning strategy, it is ensured that the agent behavior satisfies the LTL formula without relying on the environment action. The GR(1) specification accurately describes the system's behavioral requirements by separating the system and environment properties into initial conditions, safety guarantees, and liveness guarantees. It can be used to model the environment and task specifications of the reactive agent system, thereby clarifying the task objectives and behavioral constraints that the system should meet under different environmental assumptions.

[0053] Step 11: Construct the GR(1) specification constraint, the formula is:

[0054]

[0055] in, Indicates assumptions about the environment. Represents the assumptions about the system, including initial conditions, safety, and activity; if the environment can satisfy the specification described by the LTL formula, then the system can also satisfy this specification; A Boolean formula representing the initial state; The union of LTL formulas representing system invariants, Boolean formulas representing system transformation relationships; represents the union of LTL formulas with active strategies, represents a Boolean formula; express It holds true at all times; express It will always be true at some point in the future.

[0056] Step 12: Use fixed point calculation to reversely iterate from the target state to calculate the set of predecessor states that can reach the target state, and construct a synthetic strategy automaton Where Q is the finite state set, ∑ is the action alphabet, δ is the transition function, q0 is the initial state, and F is the set of accepting states.

[0057] Fixed point calculation plays an important role in transforming the liveness property in GR(1) specification into a reachability problem. First, the liveness property of the system Initialization Satisfaction The state set That is, from the state space Find satisfaction in state; the active properties of the environment Initialize the state set that does not satisfy this property That is, from the state space Find dissatisfaction in state; then for each state index i, j, calculate That is, find the In the case of s The state set of states in [i][j], where represents the entire state space, that is, the set of all possible system states; is a Boolean function, indicating that the state s satisfies the invariant of the system Indicates that there is a successor state s′ that belongs to the state set G s[i][j]; The function of the whole formula is to determine which states s satisfy the system invariant in the current iteration and can reach the reachable state set G determined in the previous iteration through a one-step transfer. s A state s′ in [i][j]; state s will be added to the state set G of the next iteration s [i][j+1]; similarly, calculate That is, find the In the case of e [i][j] is the state set of states; then update j = j + 1, and finally make a convergence judgment, repeating the above iterative calculation steps until there is G for all i s [i][j+1]=G s [i][j] and G e [i][j+1]=G e [i][j], at this time the fixed point is reached and the iteration ends.

[0058] Table 1: Reachable set G s and G e Fixed point calculation algorithm

[0059]

[0060] The present invention is based on the synchronous decentralized training paradigm of the GR(1) specification, transforms the strategy synthesis problem into a reachability problem, and solves it through fixed point calculation, decomposing the synthetic strategy automaton into individual strategy automata to guide distributed learning; this approach solves the challenge of adapting to complex dependencies in multi-agent reinforcement learning, especially when dealing with non-Markov tasks, and improves the adaptability and scalability of the system, while traditional methods are often limited in performance when facing such tasks.

[0061] Step 13: Based on the fixed point set G s and G e And the invariant and Building a synthetic strategy automaton

[0062] First, initialize the state set Q, the transition function δ and the acceptance state set F, and add the initialization state q0 to the to-be-processed set next;

[0063] Then, in the main loop, a state s is taken from next and the state index (i, j) of s is determined; if j = 0, it means that the current state satisfies the activeness goal of the agent system, that is, one of the goals in the GR (1) specification: activeness, safety; the next system goal to be verified is obtained and j is updated; for each state that satisfies the invariant and The potential successor state s′ of i is checked to see whether s′ belongs to the specified set G of i in j. s or G e set; if so, calculate the transition action a and record the transition δ(s,a)←s′; if s′ is already in Q, reuse it; otherwise, add s′ to next; if no valid successor state s′ is found when j≠0, terminate, indicating that there is no strategy for the given GR(1) specification;

[0064] Finally, the constructed synthetic strategy automaton is returned It is defined by the state set Q, the transition function δ, the initial state q0 and the acceptance state set F.

[0065] Table 2: Construction GR(1) synthesis strategy algorithm

[0066]

[0067] Compositional Strategy Automata The GR(1) strategy synthesis algorithm ensures that the constructed synthetic strategy automaton satisfies the specified invariants and goals, and the entire GR(1) synthesis process can be solved in polynomial time in terms of the state space size and the number of target attributes.

[0068] Step 2: Decompose the composite strategy automaton into individual strategy automatons, build an independent decision-making framework for each agent, reduce the complexity of coordination, enable each agent to act based on its own local observations and decisions, and improve the scalability of the system;

[0069] Composition-based strategy automata and each agent’s local action ∑ i , i∈N, construct the i-th individual strategy automaton is the finite state set of the i-th automaton, is the finite alphabet of local actions of the i-th automaton, δ i :Q i ×∑ i →Q i is the transition function of the i-th automaton, defined as:

[0070]

[0071] Among them, q 0,i ∈Q i is the initial state of the i-th automaton, if q0∈Q i , then q 0,i =q0; is the set of accepting states of the i-th automaton.

[0072] According to the decomposed strategy, each agent selects an action in the current state; each agent observes its current local environment state and selects an action, and then the system updates the state according to the actions selected by all agents and enters the next state; this process is repeated continuously to form a state sequence, simulating the evolution of the system over time; the forward simulation drives the system state transition based on the agent strategy, and verifies the effectiveness of the strategy combination by checking whether the state sequence generated during the simulation meets the safety and activity properties.

[0073] Synthetic Strategy Automata Defines how the agent system transitions between states in response to actions, ensuring that the agent system's behavior complies with the specified liveness properties of the GR(1) specification when operating under assumed environmental conditions; however, the decomposed synthetic policy automaton The parallel combination of the decomposed individual strategies may not be equivalent to the original strategy; therefore, in order to ensure the consistency of the overall intelligent system, the decomposed individual strategies are combined in parallel, and the parallel combination automaton Among them, Q || =Q1×Q2×…×Q N is the Cartesian product of the state sets of the factorized automata; is the union of the decomposition automaton alphabets; δ || :Q || ×∑ || →Q || is the transfer function, defined as: δ || ((q1,q2,…,q N ),σ)=(δ1(q1,σ),δ2(q2,σ),…,δ N (q N ,σ)), when Time δ i (q i ,σ)=q i ;q 0,|| =(q 0,1 ,q 0,2 ,…,q 0,N ) is the initial state, F || =F1×F2×…×F N is the Cartesian product of accepting states.

[0074] Through parallel combination, the strategies of each agent can work together at the agent system level; each agent makes decisions within its own state space and action range, but the combined behavior of the agents is realized through The definitions of the transfer function, initial state, and accepting state are coordinated.

[0075] For the comprehensive strategy synthesized from GR(1), the present invention decomposes the comprehensive strategy into multiple reward automata to help a single agent learn; for each i, given the local action set ∑ i ; When the agent performs an action, the state experienced by the agent and the events in the environment are merged according to certain rules; the reward automaton is constructed based on the synthetic strategy automaton of the GR(1) specification, and its structure ensures that the agent's behavior conforms to the GR(1) specification; at the same time, the effectiveness of the decomposed reward automaton is verified by using forward simulation and parallel combination.

[0076] Step 3: Construct a decentralized experience replay Q-learning algorithm with synchronization and decomposition strategies to improve the learning efficiency of multiple agents;

[0077] Step three specifically includes:

[0078] Step 31: Expand on the standard Dec-POMDP model, which is defined as Where N = {1, ..., n} represents the set of agents in the environment, each of which has a unique identifier; S represents the set of environmental states, s0 is the initial state; (O i ,A i ,R i ) is the observation-action-reward triplet of agent i, O i is a partial observation of S, A i is the action set, R i ∈O i ×A i →[0,1] defines the reward function of agent i; T∈S×A1×…×A n →Dist(S) is the state transition probability distribution, which indicates the transition probability of the environment state after the agent takes a joint action; is the global reward function; γ is a discount factor that measures the relative importance of immediate rewards versus future rewards.

[0079] The extended Dec-POMDP model is Among them, N, S, s0, (O i ,A i ), T, R, γ are defined the same as in the standard Dec-POMDP; the new parameters include: for each agent i, It is an individual strategy automaton obtained by decomposing the composite strategy of GR (1) specification, which represents the decomposed strategy of agent i and is used to guide the behavior decision of each agent, so that the agent can act based on local information and its own strategy, thereby reducing the complexity of multi-agent coordination and improving the scalability of the system; is an action labeling function, which is used to map the transitions in the Dec-POMDP to the actions in the automaton. Through this function, the agent can determine the corresponding actions to be taken in the individual policy automaton based on the observations and action selections in the Dec-POMDP environment, thereby achieving effective interaction between the high-level synthetic strategy and the low-level Dec-POMDP environment, making the agent's behavior more consistent with the overall policy requirements; is a state labeling function that maps the state of the Dec-POMDP to the state in the automaton, which helps the agent locate its own state in the individual policy automaton in the Dec-POMDP environment, so that it can make decisions according to the state transition rules of the automaton, enabling the agent to switch between state representations at different levels and better understand and follow the synthesis strategy; i ∈Q i →[0,1] is the potential function defined on each state of the automaton; these potential values ​​are calculated by the reward shaping algorithm of value iteration, reflecting the potential value of the state in the long-term learning process; in the reward calculation, the potential value is used to guide the agent to act towards a state with higher potential value, accelerate the process of the agent learning the optimal strategy, and help the agent better understand the importance of different states to achieving the ultimate goal, so as to make more favorable decisions.

[0080] Step 32: Reward shaping plays a key role in reward automata because it provides additional structure to the reward signal, thereby accelerating the learning process and more effectively guiding the agent to the optimal policy; for each state index i, the individual policy automaton All states q∈Q i , the potential value of q i (q) Initialize to 0; Enter the iteration loop and first set the variable Δ to 0. Δ is used to track the maximum change in the potential value in each iteration to determine whether the algorithm has converged;

[0081] For each state q∈Q in the automaton for each i i , do the following:

[0082] Record the potential value of the current state q as Ψ old , that is, old =Ψ i (q);

[0083] By maximizing the expected reward plus all possible local actions σ∈∑ i The discounted future potential value is used to update the potential value Ψ of the current state q i (q), the specific calculation method is:

[0084]

[0085] Among them, R i (q,σ) is the immediate reward for taking action σ in state q; γ v is a discount factor that weighs the importance of future potential value and immediate reward; i (q,σ,q′) represents the probability of taking action σ in state q to transition to state q′;

[0086] Calculate the update amount of the potential value of the current state |Ψ i (q)-Ψ old | and compare it with Δ. If the update amount is greater than Δ, then update Δ to the update amount, that is, Δ←max(Δ,|Ψ i (q)-Ψ old |); This step ensures that Δ always records the maximum change in the potential value in this iteration;

[0087] After each iteration, check whether Δ is less than the pre-set threshold ∈ v ; if Δ<∈ v , it means that the change of the potential value is small enough and the algorithm has converged, so the iteration stops; otherwise, it returns to the step of iteratively updating the potential value to continue the next iteration.

[0088] Table 3: Reward shaping algorithm based on value iteration

[0089]

[0090] Step 4: Strategy execution;

[0091] In response to the complex dependencies and coordination problems faced by intelligent agents in non-Markov tasks, the present invention constructs a decentralized experience replay Q-learning algorithm for synchronization and decomposition strategies, namely DER-QL. The synchronous automaton buffers the strategies learned from high-level synthesis strategies, thereby guiding the learning of low-level MARL (multi-agent robust chemistry system).

[0092] DER-QL builds include:

[0093] Initialize the Q value of each agent and define the initial state s and automaton state q for each agent i ; In each segment, the state is reset to the initial state s0, and this process continues until the end of the segment;

[0094] Each agent observes its current local environment state o i ; and select action a from the ε-greedy policy derived from the agent’s Q-table i ; If there are overlapping local actions between any pair of agents, synchronization is performed on the agents’ policies; for example, two individual policy automata and If they are synchronized on a certain action, then they can both enter the next state;

[0095] Update the automaton state q i and potential function Ψ i , and calculate the reward R i , that is, the data involved in a series of processes in which the agent performs actions in the environment and generates state transitions, stores the data in the replay buffer, and uses the Q learning update rule to update the Q table;

[0096] Then, the state s and observations o as well as the state q are updated for the next iteration;

[0097] Execute a set of selected actions and observe the next state s′; this process is iterative and aims to converge to the optimal strategy for each agent, facilitating effective coordination and decision-making in complex multi-agent environments.

[0098] Table 4 Decentralized experience replay Q-learning algorithm with synchronization and decomposition strategies

[0099]

[0100]

[0101] The decentralized experience replay Q-learning algorithm (DER-QL) of the synchronization and decomposition strategy of the present invention has a synchronous reward structure and can effectively use high-level synthesis strategies to guide low-level learning; through experience replay technology, the algorithm improves the stability of learning, reduces the volatility of intelligent agent decision-making, overcomes the shortcomings of traditional decentralized Q-learning in coordination and communication, and thus improves the overall learning effect.

[0102] Specific example:

[0103] like Figure 1 As shown, the industrial maintenance robot performs inspection and maintenance on the factory. This embodiment uses three agents for illustration; the three agents are divided into an inspection team and a maintenance team, agent 1 is the inspection team, and agents 2 and 3 are the maintenance team; the agents move and cooperate based on the GR (1) protocol to complete the overall task; the state space of the agents completing the assembly task is set to be discrete, while the action space of the agents is set to be continuous; regarding the state space, the entire warehouse is decomposed into multiple grids; as shown in FIG. Figure 1In the lower left figure, different areas are divided into inspection team-related areas and maintenance team-related areas, where A, B, and C are inspection team-related areas, A represents the initial position of agent 1, area B is the area where agent 1 goes after completing the task in area A, and C is the task target area of ​​agent 1. Arriving here means that the inspection task is completed; D, E, F, G, H, and I are maintenance team-related areas, area D is the initial position of agent 2, area E is the area where agent 2 goes after completing related operations in area D, and area F is the task target area of ​​agent 2, which is also one of the prerequisites for agent 3 to enter the final maintenance area I; area G is the initial position of agent 3, area H is the area where agent 3 goes after completing related operations in area G, and area I is the task target area of ​​agent 3, that is, the final maintenance location; only when agent 1 reaches area C and agent 2 reaches area F, can agent 3 enter this area to perform the final maintenance task.

[0104] The system constraints of the three robots generated by GR(1) can be expressed as follows:

[0105] ENV: ab c;

[0106] SYS:de fg hi;

[0107] ENVINIT:!a&!b$!c;

[0108] ENVTRANS:[](a->b')&[](b->c')&[](!(a'&b'))

[0109] &[](!(a'&c'))&[](!(b'&c'));SYSINIT:! y&! z;

[0110] ENVGOAL:[]<>c;

[0111] SYSINIT:[](d->e')&[](e->f')&[](g->h')&[](c&f&h->i')

[0112] &[](!(f'&d'))&[](!(f'&e'))&[](!(d'&e'))

[0113] &[](!(g'&h'))&[](!(g'&i'))&[](!(h'&i'));

[0114] SYSGOAL:[]<>(f&i);

[0115] At the beginning, agent 1 is initially in area A, which represents the inspection area. In the initial state, areas B and C are not occupied, and agent 1 is in a waiting state, ready to start the inspection task; agent 2 starts in area D, which is its initial standby position; agent 3 is initially in area G, and in the initial state, areas H and I are not occupied, and agent 3 is ready to start collecting spare parts; when the task starts, agent 1 moves from area A to area B, completes the inspection of the machine in area B and performs maintenance preparations, such as checking the various parameters of the machine, recording the equipment status, etc.; then, agent 1 moves from area B to area C, and arriving at area C indicates that the inspection task is completed, and at the same time sends a signal to the maintenance team that the machine is ready for maintenance. At this time, agent 1 waits in area C for subsequent instructions or continues to monitor the machine status.

[0116] Agent 2 moves from area D to area E, where it collects tools needed to maintain the machine, such as wrenches and screwdrivers, to ensure the integrity and availability of the tools. Agent 2 then moves with the collected tools from area E to area F (tool delivery location), waiting to deliver the tools to Agent 3 who is responsible for final maintenance. In area F, Agent 2 ensures that the tools are ready for delivery at any time and maintains communication with Agent 3 to prepare for tool handover.

[0117] Agent 3 moves from area G to area H, where it collects spare parts, such as key components of the machine, to ensure the quality and adaptability of the parts. When agent 1 arrives at area C and agent 2 arrives at area F, agent 3 moves from area H to area I (the final maintenance location), where it uses the tools provided by agent 2 and the parts it has collected to perform final maintenance on the machine, such as replacing damaged parts and adjusting equipment parameters. After completing the maintenance, agent 3 can return to its initial position or continue to stand by according to the task requirements to prepare for the next maintenance task.

[0118] A smart factory is a modern factory that uses advanced information technology and manufacturing technology to achieve intelligent production, management and services. Traditional industrial maintenance relies on manual labor, with workers regularly checking and maintaining equipment. This approach may lead to unnecessary downtime or maintenance before the equipment fails. In a smart factory, linear temporal logic is used to describe the specifications of normal equipment operation and possible failure modes. Possible equipment failures can be predicted in advance, and maintenance can be arranged in a timely manner to reduce equipment downtime and improve production efficiency.

[0119] Aiming at the problem of multi-agent industrial maintenance, the present invention proposes a multi-agent industrial maintenance method based on linear temporal logic, and decomposes the proposed synthesis strategy into multiple reward automata for the agents to learn, thereby improving the overall training speed; the present invention also proposes to apply the decentralized experience replay Q learning algorithm with synchronization and decomposition strategies to the multi-agent industrial maintenance process. In order to verify the method of the present invention, Figure 2 As shown, the training effects of the method DER-QL of the present invention are compared with five existing methods: decentralized Q-learning based on projection reward mechanism (DQPRM), centralized experience replay Q-Learning (CER-QL), decentralized Q-Learning (DQL), hierarchical reinforcement learning based on reward machine (HierarchicalQL), decentralized Q-learning and reward shaping based on independent reward machine (DQ-iRM+RS); the experimental results show that: compared with other methods, the method of the present invention can enable the robot group to learn the optimal strategy faster and also obtain higher cumulative rewards.

[0120] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A strategy generation and decomposition method based on linear temporal logic control, characterized in that: The following steps are involved: Step 1: Use GR(1) to regulate the behavior of several agent systems, formalize the strategy synthesis into reachability, calculate the fixed point set to determine the state that satisfies the agent's goal, and construct an automaton to implement the GR(1) specification so that the agent satisfies the safety and liveness properties under the assumptions of the environment; Step 2: Decompose the composite strategy automaton into individual strategy automatons, so that each agent takes actions based on its own local observations and decisions; Step 3: Construct a decentralized experience replay Q-learning algorithm with synchronization and decomposition strategies; Step 4: Use the DER-QL algorithm to synchronize the automaton buffer with the strategies learned from the high-level synthesis strategy, thereby guiding the learning of low-level MARL.

2. The strategy generation and decomposition method based on linear temporal logic control according to claim 1 is characterized in that: Step 1 specifically includes: Step 11, construct GR(1) specification constraints; Step 12: Use fixed point calculation to reversely iterate the target state to calculate the set of predecessor states to reach the target state, and construct a synthetic strategy automaton Where Q is a finite state set, ∑ is the action alphabet, δ is the transition function, q0 is the initial state, and F is the set of accepting states; Step 13: Based on the fixed point set G s and G e And the invariant and Building a synthetic strategy automaton 3. The strategy generation and decomposition method based on linear temporal logic control according to claim 2 is characterized in that: Fixed point calculations include: Initialization satisfies the activity properties of the system The state set Initializes liveness properties that do not satisfy the environment The state set For each state index i, j, calculate Represents the entire state space; is a Boolean function, indicating that the state s satisfies the invariant of the system Indicates that there is a successor state s′ that belongs to the state set G s [i][j]; calculate Update j = j + 1 and perform convergence judgment until G s [i][j+1]=G s [i][j] and G e [i][j+1]=G e [i][j], the fixed point is reached and the iteration ends.

4. The strategy generation and decomposition method based on linear temporal logic control according to claim 2 is characterized in that: Step 13 specifically includes: Initialize the state set Q, the transition function δ and the acceptance state set F, and add the initialization state q0 to the pending set next; Take a state s from next and determine the index of s (i, j); if j = 0, get the next system goal to be verified and update the state index j; For each invariant and The potential successor state s′ of i is checked to see whether s′ belongs to the specified set G of i in j. s or G e In the set; if so, calculate the transfer action a and record the transfer δ(s,a)←s′; if s′ is already in Q, reuse it; otherwise add s′ to next; If no successor state s′ is found when j≠0, terminate; return to the synthetic strategy automaton 5. The strategy generation and decomposition method based on linear temporal logic control according to claim 1 is characterized in that: Step 2 specifically includes: Constructing individual policy automata and parallel combinatorial automata Among them, Q || =Q1×Q2×…×Q N is the Cartesian product of the states of the factorized automaton, is the union of the decomposition automaton alphabets, δ || :Q || ×∑ || →Q || is the transfer function, defined as: δ || ((q1,q2,…,q N ),σ)=(δ1(q1,σ),δ2(q2,σ),…,δ N (q N ,σ)), when Time δ i (q i ,σ)=q i ;q 0,|| =(q 0,1 ,q 0,2 ,…,q 0,N ) is the initial state, F || =F1×F2×…×F N is the Cartesian product of accepting states.

6. The strategy generation and decomposition method based on linear temporal logic control according to claim 1 is characterized in that: Step three specifically includes: Step 31: Construct the extended Dec-POMDP model. N represents the set of agents in the environment; S represents the set of environmental states, s0 is the initial state; (O i ,A i ,R i ) is the observation-action-reward triplet of agent i; T is the state transition probability distribution; R is the global reward function; γ is the discount factor; is the individual strategy automaton; is the action labeling function; is the state labeling function; Ψ i is the potential function on each state of the automaton; Step 32: For each state index i, The potential value of state q is Ψ i (q) is initialized to 0; enter the iteration loop and set the variable Δ that tracks the maximum change in the potential value in each iteration to 0; Record the potential value of the current state q as Ψ old , updates the potential value Ψ of the current state q by maximizing the expected reward plus the discounted future potential value of all possible local actions σ i (q); Calculate the update amount of the potential value of the current state |Ψ i (q)-Ψ old |, and compare it with Δ. If the update amount is greater than Δ, then Δ←max(Δ,|Ψ i (q)-Ψ old |); After each iteration, check whether Δ is less than the preset threshold ∈ v ; if Δ<∈ v , the iteration stops; otherwise, return to the step of iteratively updating the potential value to continue the next iteration.

7. The strategy generation and decomposition method based on linear temporal logic control according to claim 1 is characterized in that: The construction of the DER-QL algorithm includes: Initialize the Q value of each agent, define the initial state s of the agent and the state q of the automaton i ; The agent observes the local environment state o i ; Select action a from the ε-greedy strategy in the Q table i ; If there are overlapping local actions between any pair of agents, synchronization is performed on the agents’ policies; Update the automaton state q i and potential function Ψ i , and calculate the reward R i , update the Q table using the Q learning update rule; Update the state s, observation value o and state q for the next iteration; Execute the selected action set a and observe the next state s ′ .

8. A strategy generation and decomposition system based on linear temporal logic control, characterized in that: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the strategy generation and decomposition method based on linear temporal logic control as described in any one of claims 1 to 7.

9. A computer readable medium storing a computer program code, characterized in that: When the computer program code is executed by a processor, the strategy generation and decomposition method based on linear temporal logic control as described in any one of claims 1 to 7 is implemented.

Images