A method for generating a maximum permissive coordinator of a discrete event system

Abstract

The application discloses a method for generating a maximum allowable coordinator of a discrete event system, comprising the following steps: 1) defining a model; 2) training of a sub-coordinator; 3) sampling and verification of the sub-coordinator; and 4) merging of the sub-coordinators. The method can obtain a coordinator satisfying maximum allowance, can improve the training speed of an intelligent agent, and can terminate the training in advance without waiting for complete convergence of a reward function.

Description

Technical Field

[0001] This invention relates to supervisory control technology for discrete event systems such as automated guided vehicle (AGV) systems, multi-robot collaboration, and traffic control. In particular, it relates to a design technology that uses Deep Reinforcement Learning (DRL) to generate sub-coordinators and achieves maximum permissibility coordinator through policy merging. Specifically, it is a method for generating maximum permissible coordinators for discrete event systems. Background Technology

[0002] Discrete Event Systems (DES) are a class of dynamic systems whose state transitions are driven by discrete events. They are widely used in industrial automation, such as power dispatching networks, traffic control systems, communication protocol stacks, and multi-robot collaborative systems. In the field of supervisory control of DES, existing technologies mainly employ global control and modular control.

[0003] In global control, Supervisory Control Theory (SCT) constructs a global system model and generates optimal controllers to ensure that system behavior meets control requirements. However, due to the need to calculate the synchronization product of the global system, this method faces the state explosion problem when dealing with multi-module systems, with computational complexity increasing exponentially with the number of system components. To alleviate this problem, modular supervisory control decomposes the global specification to generate independent modular controllers, significantly reducing computational overhead. However, each modular controller makes independent decisions based on a local perspective, lacking global information, which easily leads to control conflicts, causing the system to deadlock or become blocked. Although introducing a coordinator can alleviate conflicts, generating the coordinator still requires calculating the synchronization product of each modular controller, and the state explosion problem persists. In recent years, Dynamic Management Link (DRL) has provided a new approach to solving the supervisory control of multi-module systems. Its approach represents the global state of the system as agent observations and a subset of controllable events as actions. A reward function guides the agent to learn the set of events allowed to be executed in each state. This method eliminates the need to construct the synchronization product of each modular controller, thus reducing the computational overhead of coordinator generation and mitigating the state explosion problem to some extent. However, in existing research, the control policies trained by DRL still have significant limitations in terms of the maximum permissibility of system behavior: 1) Failure to meet maximum permissibility: The stochastic exploration mechanism (such as the ε-greedy policy) on which the DRL algorithm relies is difficult to fully learn all the events that should be allowed in each state within a limited training time. In addition, in order to accelerate convergence, the action space of the agent is restricted. These factors lead to the coordinator policy learned by DRL only allowing some system behaviors, thus making it difficult to meet the maximum permissibility; 2) High-dimensional state space training challenge: In discrete event systems, especially in large-scale scenarios, the system state space has a high dimension. When DRL uses neural networks as function approximators to process such high-dimensional discrete state spaces, training convergence is difficult. Therefore, under the premise of avoiding the state space explosion caused by synchronous product calculation, designing a coordinator generation method that can overcome the above-mentioned limitations of DRL, thereby effectively ensuring that the system behavior meets the control requirements and significantly expanding the range of system behaviors allowed by the coordinator, is a complex and urgent technical challenge in this field.

[0004] The relevant theoretical definitions in the existing technology are as follows: alphabet A string is a finite set of non-empty events, and a finite sequence of characters in the alphabet is called a string. An empty string is denoted as . ;gather It is defined in The Kling closure on the string consists of all strings of finite length and an empty string; for strings... , and Connection It is indicated, abbreviated as ;language It is defined in A set of strings above, that is Given two languages The connection between two languages ​​can then be defined as ;language The prefix closure is denoted as , ;if So, what is language? It is prefix closed; Discrete event systems can be handled by deterministic finite state automata. It means that, among them, A set of states; For the set of events of an automaton; This is the state transition function; This is the initial state; For a set of labeled states, a finite state automaton The generated language is denoted as Defined as: , indicating from the initial state Starting from this point, the sequence of all possible events that the system can execute is denoted in system markup language as follows: Defined as This represents the sequence of events that enables the system to complete the task. Obviously, ; Discrete event systems The set of events can be divided into: , A set of controllable events For a language, it is a set of uncontrollable events. ,like satisfy: Then it is called language It is about Controllable, and further, for a given language When language Simultaneously satisfying: (1) Controllability condition: (2) Closure condition of markup language: , It is controllable and non-blocking; For automatic machines If a string exists Make Then it is called a state. From the initial state Reachable if a string exists Make Then it is called a state. Trim property: If every state in an automaton is both reachable and trim-reachable, then the automaton is said to satisfy the reachability and trim property. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for generating a maximum permissible coordinator for discrete event systems. This method can obtain a coordinator that satisfies maximum permissibility, improves the training speed of agents, and allows training to terminate early without waiting for the reward function to fully converge.

[0006] The technical solution to achieve the objective of this invention is: A method for generating a maximum permissible coordinator in a discrete event system includes the following steps: 1) Define the model: Discrete event system Discrete event system Depend on A locally controlled system The composition, and the corresponding state sets are respectively Meanwhile, the system has Module controller The state sets are respectively The global state space is defined as a vector consisting of the states of the locally controlled system and the states of the module controller. ,in To ensure consistency in state identification during subsequent training, validation, and sub-coordinator merging, for two global states... If satisfied and Two states are considered the same state, that is ; 2) Training of sub-coordinators: Construct a training environment based on deep reinforcement learning (DRL), using a set of controllable events as the action space, and employing a reward function to guide the agent in learning a coordination strategy that meets control requirements. Specifically, the system state is represented as a global state vector composed of the states of each locally controlled system and the states of the module controllers. The global state represents the current overall operating status of the controlled system, and the agent is in each state. Select action ,in This represents a subset of controllable events selected from the set of controllable events, based on the action. The set of events that can be executed can be determined. ,in, This indicates the global state of the controlled system. The actual set of executable events, therefore, The events in the data are still a collection of system events. Events in the event In state Whether execution is allowed depends on simultaneously satisfying the constraints of the current state of the controlled system and the current state of the module controller. It can be seen that uncontrollable events only require the state to be controlled. Once it is actually executable, it will be kept in the collection. In this way, it is ensured that the sub-coordinator will not disable any uncontrollable events, assuming Let be the state transition function of the controlled system in the global state space, when in state From set Select Event Upon execution, the system transitions to the next state. Rewards are given based on the state transition results, and a one-step reward can be recorded as... , , Indicates event reward, Indicates the reward for the next state; the next global state. A positive reward is given when a state belongs to the marked state set; a negative reward is given when the next global state is deadlocked; and zero reward is given for other state transitions. By maximizing the cumulative reward, the agent gradually learns the optimal control strategy. The training of the sub-coordinator adopts the following algorithm: Algorithm 1: Sub-coordinator Training Algorithm Based on Deep Reinforcement Learning Input: Controlled system , Maximum number of training rounds Maximum number of moves per round ; Output: Optimal Strategy ,include: 1-1) Initialization Network parameters and initialize the target network parameters. ; 1-2) Initialize experience pool B; 1-3) For to do; 1-4) Initialize the system state to the initial global state. ; 1-5) For to do; 1-6) According to Strategy selection action ; 1-7) Calculate the set of events that are allowed to be executed in the current state. ; 1-8) If then 1-9) Order ; 1-10) Give negative rewards ; 1-11) Else; 1-12) From the set Select Event ; 1-13) Execution state transition ; 1-14) Calculate the reward based on the state transition result. ; 1-15) End If; 1-16) Experience Store in experience pool ; 1-17) Randomly sample small batches of experience from the experience pool; Updated 1-18 Network parameters ; 1-19) Update the target network parameters according to the set period. ; 1-20) If Then, the state becomes deadlocked. 1-21) End the current training round; 1-22) End If; 1-23) Order ; 1-24) End For; 1-25) End For; 1-26) Based on the training The network obtains the optimal strategy ; 3) Sampling and Verification of Sub-Coordinators: During reinforcement learning training, the system saves the policy model in each training round, forming a policy set. Since reinforcement learning training has phased characteristics, the policies obtained at different training stages may differ in their selection of controllable events under the same state. Therefore, sparse and uniform sampling is performed from the policy set at fixed intervals. The trained policy function is then evaluated by traversing the global state space. The policy outputs the corresponding actions for each state, and the actions determine the set of events allowed to be executed. Based on this, the state and transition relationships of the sub-coordinators are constructed, resulting in the sub-coordinator set. ,in To determine the number of sub-coordinators obtained through sampling, specifically, a maximum number of training epochs can be set as the training upper limit, and a target sampling number threshold can be set. During the sampling process, each policy model collected is counted once. When the cumulative sampling number reaches the target threshold, the DRL training and sampling process can be terminated. The termination condition does not depend on the complete convergence of the reward function, but only on the fact that a sufficient number of policy models have been obtained, thereby shortening the training cycle and reducing computational overhead, and obtaining the original set of sub-coordinators. Then, non-blocking verification is performed on each sub-coordinator, and Algorithm 2 is used to test the sample set. Each sub-coordinator in the algorithm is verified, and valid sub-coordinators that satisfy the Trim property are selected to obtain the set of verified sub-coordinators. , Algorithm 2 is the specific verification process for the sub-coordinator: Algorithm 2: Sub-coordinator Verification Algorithm Input: Sub-coordinators obtained from DRL Controlled system ; Output: Verified sub-coordinators Or NULL, including: 2-1) Initialize the reachable state set Initialize queue ; 2-2) While queue Non-empty do; 2-3) Retrieve the current state from the queue ; 2-4) Initialize the set of executable events in the current state. ; 2-5) For each uncontrollable event do; 2-6) If Defined and Then is not defined; 2-7) Return NULL; 2-8) Else If Defined and There is a definition for then; 2-9) join in ; 2-10) End If; 2-11) End For; 2-12) For each controllable event do; 2-13) If Defined and There is a definition for then; 2-14) join in ; 2-15) End If; 2-16) End For; 2-17) If and then; 2-18) Returns NULL; 2-19) End If; 2-20) For each do; 2-21) Calculate the next state ; 2-22) If state Then was not visited; 2-23) join in and joined the team; 2-24) End If; 2-25) End For; 2-26) End While; 2-27) Initialize the common reachable state set Initialize the reverse queue ; 2-28) While queue Non-empty do; 2-29) Retrieve the state from the queue ; 2-30) For each state do; 2-31) For each event do; 2-32) If and then; 2-33) will join in Merge into the reverse queue; 2-34) End If; 2-35) End For; 2-36) End For; 2-37) End While; 2-38) If| |=| | then; 2-39) Return Verification passed; 2-40) Else; 2-41) Returns NULL, indicating a livelock exists; 2-42) End If; 4) Merging of sub-coordinators: Obtaining the set of verified sub-coordinators. Next, multiple sub-coordinators need to be merged, and the final coordinator is constructed by performing a union operation on the sub-coordinators: Let the sub-coordinators be... The merge coordinator is defined as follows: , where the state set , mark state set transfer function For any If it exists Make ,but The merging process of the sub-coordinators adopts Algorithm 3: Algorithm 3: Sub-coordinator merging algorithm Input: The set of verified sub-coordinators ; Output: The coordinator obtained by merging ,include: 3-1) Initialization: Let transfer function Empty; 3-2) For each sub-coordinator do; 3-3) For each state do; 3-4) If state (This state is already in the state set) (in Chinese) then; 3-5) For each event do; 3-6) If Defined and Then is not defined; 3-7) Order and order ; 3-8) If state then; 3-9) join in ; 3-10) If state then; 3-11) will join in ; 3-12) End If; 3-13) End If; 3-14) End If; 3-15) End For; 3-16) Else, state ; 3-17) will join in ; 3-18) If state then; 3-19) join in ; 3-20) End If; 3-21) End If; 3-22) End For; 3-23) End For; 3-24) Output Coordinator .

[0007] Among them, step 1-1) initialization Network parameters and initialize the target network parameters. Steps 1-2) initialize the experience pool B to store the experience data generated during training; Steps 1-3) to 1-25) constitute the reinforcement learning training process, where the number of training rounds is set in step 1-3). Then begin the training loop; in steps 1-4), initialize the system state to the initial global state. Steps 1-5) to 1-24) constitute the training process within a single training round; in step 1-6) according to Strategy selection action This is used to weigh the trade-offs between exploration and exploitation; steps 1-7) based on the event set function. Calculate the set of events allowed to be executed in the current state; in steps 1-8) to 1-15), if This indicates that there are no executable events in the system under the current action, and the system state remains unchanged. And give a negative reward; otherwise, remove from the event set. Select Event And according to the state transition function Next state of the computing system Calculate the reward based on the state transition result. Steps 1-16) to 1-19) are used for experience replay and network parameter updates; specifically, in step 1-16), the experience... Store in experience pool B; Step 1-17) Randomly sample a small batch of experience from the experience pool; Step 1-18) Update the Q network parameters using the sampled experience; Step 1-19) Update the target network parameters according to the set period; In steps 1-20) to 1-22), determine whether a deadlock state has been reached. If a deadlock state is reached, terminate the current training round; Step 1-23) Update the system state to... And continue training for the current round; when the maximum number of training rounds is reached... Then, the algorithm ends; in steps 1-26), the optimal strategy is determined based on the trained Q-network. This allows us to obtain the actions selected in each global state, and based on this, determine the set of events that the system is allowed to execute, thus forming a sub-coordinator; Steps 2-1) to 2-26) are used to calculate the reachable state set of the sub-coordinator; Step 2-1) initializes the reachable state set and the search queue, and uses the initial state of the sub-coordinator. As the initial state; steps 2-2) to 2-26) use a breadth-first search to traverse the reachable states of the sub-coordinators; in steps 2-4) to 2-16), construct the set of executable events in the current state. For uncontrollable events If the controlled system In state The sub-coordinator allows this event to occur. The event transfer is not defined, i.e. Defined and If undefined, it means that the sub-coordinator has disabled uncontrollable events, and it will directly return NULL; if If both are defined, then add the event to the set. For controllable events, only when and Only when all elements are defined are they added to the set. Therefore, set This represents the set of events that are actually executable in the current state; steps 2-17) to 2-19) are used to detect deadlock, if the current state is an unmarked state and If the system cannot continue executing any events in this state, a deadlock is determined and NULL is returned; Steps 2-20) to 2-25) traverse the set. The events in the process are recorded, and the next state is calculated based on the state transition function of the sub-coordinator. The newly discovered state is added to the reachable state set; steps 2-27) to 2-37) are used to calculate the common reachable state set, and this process is used to label the state set. Starting from the beginning, perform a reverse search to find all states that can reach the marked state; finally, steps 2-38) to 2-42) determine whether all reachable states are mutually reachable states. If they satisfy | |=| If |, it means that the sub-coordinator satisfies the Trim property, the verification passes and the sub-coordinator is returned; otherwise, it means that a livelock exists, and NULL is returned. Step 3-1) Initialize the coordinator With the first sub-coordinator initial state This is the initial state; steps 3-2) to 3-23) are the sub-coordinator merging process, which iterates through the set sequentially. The states and transition relationships of each sub-coordinator are as follows: Specifically, step 3-3) traverses the current sub-coordinator. state set If the current state Already exists in the state set In the process, then in steps 3-5) to 3-15), we traverse each event in this state, and for any event... If the sub-coordinator transfers Defined and coordinator Corresponding transfer If not yet defined, the transfer will be incorporated into the coordinator. At this point, let the target state be... If the target state Not part of the state set Then add it in steps 3-8) to 3-9). If the target state The set of marked states belonging to the sub-coordinator Then, in steps 3-10) to 3-11), it is added to the marked state set. If the current state is not in the state set In steps 3-16) to 3-17), the state is directly added. If the state also belongs to the marked state set of the sub-coordinator Then, in steps 3-18) to 3-19), it is added to the marked state set. After the algorithm completes, the merged coordinator is output in step 3-24. The time complexity analysis of Algorithm 3 is as follows: outer loop (step 3-2) traversal Each sub-coordinator; the middle loop (step 3-3) iterates through the state set of each sub-coordinator. In the worst case, the number of states is The inner loop (steps 3-5) iterates through the event set for each state. The worst-case event count is The total time complexity of Algorithm 3 is .

[0008] Compared with the existing technology, the present technical solution has the following technical effects: 1. Due to the randomness of the exploration strategy and the compression of the action space, the coordinator trained by the DRL method is difficult to satisfy the maximum permissibility. This technical solution merges multiple sub-coordinators so that the merged coordinator can execute the events allowed by each sub-coordinator, thereby obtaining a coordinator that satisfies the maximum permissibility. 2. This technical solution does not require the sub-coordinator obtained in a single training session to meet the maximum allowable requirement. Therefore, by limiting the size of the action space, the training speed of the agent can be significantly improved. 3. This technical solution adopts a sparse uniform sampling method. When the number of saved strategies reaches a preset threshold, training can be terminated in advance without waiting for the reward function to fully converge.

[0009] This method can obtain a coordinator that satisfies maximum permissibility, improve the training speed of agents, and allow training to be terminated early without waiting for the reward function to fully converge. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the AGV manufacturing system structure in the embodiment; Figure 2 A graph showing the average reward curve for DQN training; Figure 3 This is a comparison chart of the number of coordinator states obtained at each sampling time. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but these are not intended to limit the scope of the invention.

[0012] Example: In this example, the manufacturing system consisting of five Automated Guided Vehicles (AGVs) is the object, and the system structure is as follows: Figure 1 As shown, the system consists of two input workstations, IPS1 and IPS2, three processing workstations, WS1, WS2, and WS3, and a finished product station, CPS. Five AGVs are responsible for loading, transporting, and unloading parts between the workstations along a fixed circular path. The difference in the input quantities of the two types of parts must meet the following constraints. The system has two module controllers, IPSR and ZWSR. IPSR is responsible for constraining the difference in the quantity of two types of parts, while ZWSR is responsible for constraining the capacity of each shared area and the occupancy of each workstation. Since each module controller makes independent decisions based on its local perspective, control conflicts will occur during operation. Therefore, a coordinator is needed to eliminate conflicts between controllers and ensure the non-blocking nature of the system without interfering with the control logic of each module controller. If the SCT method is used directly to generate the coordinator, the synchronization product of all module controllers and the controlled system needs to be calculated, which faces the state explosion problem. The goal of this example is to generate a coordinator that satisfies the maximum permissibility without performing the synchronization product calculation. Specifically: System Event Set It contains a total of 26 events: Controllable Event Set There are 10 uncontrollable events in total; There are 16 events in total, and the physical meaning of each event is shown in Table 1. The global state vector is defined as follows: ,in This shows the current status of the five AGVs. Given the current states of the two module controllers, this example also uses the SCT method to directly synthesize the maximum permissible coordinator, achieving a reachable state count of 4406. This serves as a benchmark for evaluating whether the coordinator generated by this method satisfies the maximum permissibility requirement. Table 1. Meaning of System Events ; Sub-coordinator Training: Construct a DRL training environment corresponding to the five-AGV manufacturing system and call Algorithm 1 to train the policy. In this example, the Deep Q-Network (DQN) algorithm is used as the specific implementation of DRL, with the global state vector as the basis. As the agent observes the state, a subset of controllable events is used as actions. In this example, the system has 10 controllable events, and the action space is [formula missing] when unrestricted. To reduce training difficulty, the number of controllable events allowed in each state is limited to no more than one. Therefore, the action space is reduced from the unrestricted state. A subset of controllable events is compressed into In this type, the Q-value function is approximated by a two-layer fully connected neural network (128 neurons per layer, with ReLU activation function), and the output dimension is 11. The reward design combines event-based rewards with status-based rewards, providing rewards in one step. Key events for AGV5 to complete transportation tasks Positive rewards of 1, 5, and 10 are given respectively; a penalty of -30 is given when the system enters a deadlock state or the set of allowed execution events is empty; the reward for other state transitions is 0. The main hyperparameters used for training are shown in Table 2. Table 2 DQN Training Hyperparameter Settings ; like Figure 2 As shown, Figure 2 The graph shows the average reward curve of the DQN agent during 50,000 training rounds. The vertical axis represents the moving average reward over the past 200 rounds. For the first 15,000 rounds, the reward is stable around -5, indicating the agent is in the exploratory phase. Afterward, the reward slowly increases with significant fluctuations, and converges after approximately 42,000 rounds. Policy Sampling, Sub-Coordinator Construction and Validation: During training, sparse and uniform sampling is performed from the policy set at fixed intervals. The sampling interval is set to 1000, and the target sampling number threshold is set to 30. When training reaches the 30,000th round, a total of 30 policy models are obtained, corresponding to the sampling point set. Sampling then stopped, but training continued until the 50,000th round for subsequent comparison. For each sampled policy model, values ​​were evaluated state-by-state in the global state space to determine the action output and corresponding set of allowed execution events for each state. Sub-coordinators were then constructed by combining the global state transition relationships of the controlled system, resulting in the original set of sub-coordinators. Subsequently, on Algorithm 2 is called to verify each sub-coordinator in order to filter out valid sub-coordinators that satisfy the Trim property. The verification results show that out of the 30 original sub-coordinators, 27 passed the verification and 3 failed. The main reason for the failure of the sub-coordinators is that the policy is not stable in the early stage of training, and deadlock or livelock occurs in some global states. The set of sub-coordinators that passed the verification is denoted as . Table 3 shows the number of training rounds, the number of reachable states, and the validation results for each valid sub-coordinator: Table 3. Basic parameters of each sub-coordinator that have passed verification. ; child coordinator For example, its corresponding training round number is 1000, the number of reachable states is 3512, and it passes the verification after Algorithm 2; taking the sub-coordinator corresponding to the 3000th round in the original sub-coordinator as another example, during the verification process, it was detected that the set of executable events in the unmarked state was empty, so it was judged to have deadlock and the verification failed. Sub-coordinator merging: Merging the set of verified sub-coordinators Algorithm 3 is called to merge the components, resulting in the coordinator. During the merging process, the states and state transitions of each sub-coordinator are sequentially incorporated into the merging coordinator. The number of coordinator states gradually increases with the merging process. Table 4 shows the changes in the number of states after each round of merging: Table 4 Coordinator during the merger process State number changes ; Results analysis: such as Figure 3 As shown, Figure 3 The number of reachable states of the sub-coordinators and the number of merged coordinators corresponding to the 30 sampling points are given. The number of states fluctuates between 3500 and 4100 for each sub-coordinator, showing an overall upward trend as training progresses. The number of states reachable by the merged coordinator increases with each merging step, reaching 4406 after the sub-coordinator corresponding to round 26000 is added in step 23, and then stops increasing. From step 24 to step 27, no new states are added (the number of new states is 0), and the merging process converges. To ensure a fair comparison, this example did not stop training immediately after sampling, but continued training DQN until the 50,000th round until convergence. The training curve is shown below. Figure 2 As shown, the converged neural network is represented by a global state vector. Using this as input, the Q-value is calculated state by state, and the optimal action is selected to determine the set of allowed executable events in each state. Based on this, a coordinator is constructed. The number of achievable states for the resulting coordinator is only 4025, which is lower than the SCT benchmark of 4406, failing to satisfy the maximum allowableness. This indicates that even if DQN training is fully converged, the action space of a large-scale discrete event system grows exponentially with the number of controllable events. To make training feasible, the action space must be compressed. However, the compressed action space limits the set of controllable events that the policy can allow, resulting in the convergence policy still failing to satisfy the maximum allowableness. The comparison results of the three methods are shown in Table 5. Table 5 Comparison of the three methods , The SCT method yields a coordinator with 4406 reachable states, satisfying the maximum allowable requirement, but requires synchronization product calculation; the DQN method, after convergence, constructs a coordinator with 4025 reachable states, requiring no synchronization product calculation, but does not satisfy the maximum allowable requirement; the merged coordinator generated by this example method... The number of reachable states is 4406, no synchronization product calculation is required, and the maximum permissibility is satisfied. The method in this example only uses the sampling strategy in the first 30,000 rounds for verification and merging, which has achieved the same result as the SCT method. This shows that in practical applications, there is no need to wait for DQN to fully converge. Training can be terminated early after sampling, which effectively reduces the computational overhead of the coordinator generation.

Claims

1. A method for generating a maximum permissible coordinator in a discrete event system, characterized in that, Includes the following steps: 1) Define the model: Discrete event system Discrete event system Depend on A locally controlled system The composition, and the corresponding state sets are respectively The system has Module controller The state sets are respectively The global state space is defined as a vector consisting of the states of the locally controlled system and the states of the module controller. ,in For two global states If satisfied and Two states are considered the same state, that is ; 2) Training of sub-coordinators: Construct a training environment based on deep reinforcement learning (DRL), using a set of controllable events as the action space, and employing a reward function to guide the agent in learning a coordination strategy that meets control requirements: The system state is represented as a global state vector composed of the states of each locally controlled system and the states of the module controllers. The global state represents the current overall operating status of the controlled system, and the agent is in each state. Select action ,in This represents a subset of controllable events selected from the set of controllable events, based on the action. The set of events that can be executed can be determined. ,in, This indicates the global state of the controlled system. The actual set of executable events. The events in the data are still a collection of system events. Events in the event In state Whether execution is allowed depends on simultaneously satisfying the constraints of the current state of the controlled system and the current state of the module controller. It can be seen that uncontrollable events only require the state to be controlled. Once it is actually executable, it will be kept in the collection. In this context, it is ensured that the sub-coordinator will not disable any uncontrollable events. Let be the state transition function of the controlled system in the global state space, when in state From set Select Event Upon execution, the system transitions to the next state. Rewards are given based on the state transition results, and a one-step reward can be recorded as... , , Indicates event reward, Indicates the reward for the next state; the next global state. A positive reward is given when a state belongs to the marked state set; a negative reward is given when the next global state is deadlocked; and zero reward is given for other state transitions. By maximizing the cumulative reward, the agent gradually learns the optimal control strategy. The training of the sub-coordinator uses Algorithm 1: Algorithm 1: Sub-coordinator Training Algorithm Based on Deep Reinforcement Learning Input: Controlled system , Maximum number of training rounds Maximum number of moves per round ; Output: Optimal Strategy ,include: 1-1) Initialization Network parameters and initialize the target network parameters. ; 1-2) Initialize experience pool B; 1-3)For to do; 1-4) Initialize the system state to the initial global state. ; 1-5)For to do; 1-6) According to Strategy selection action ; 1-7) Calculate the set of events that are allowed to be executed in the current state. ; 1-8)If then 1-9) Order ; 1-10) Give negative rewards ; 1-11) Else; 1-12) From the set Select Event ; 1-13) Execution state transition ; 1-14) Calculate the reward based on the state transition result. ; 1-15) End If; 1-16) Experience Store in experience pool ; 1-17) Randomly sample small batches of experience from the experience pool; Updated 1-18 Network parameters ; 1-19) Update the target network parameters according to the set period. ; 1-20) If Then, the state becomes deadlocked. 1-21) End the current training round; 1-22) End If; 1-23) Order ; 1-24) End For; 1-25) End For; 1-26) Based on the training The network obtains the optimal strategy ; 3) Sampling and Validation of Sub-Coordinators: During reinforcement learning training, the system saves the policy model in each training round, forming a policy set. Sparse and uniform sampling is performed from this policy set at fixed intervals. The trained policy function is evaluated by traversing the global state space. The policy outputs the corresponding actions for each state, and the actions determine the set of events that can be executed. Based on this, the state and transition relationships of the sub-coordinators are constructed, resulting in the sub-coordinator set. ,in To determine the number of sub-coordinators obtained through sampling, a maximum number of training epochs is set as the training upper limit, and a target sampling number threshold is also set. During the sampling process, each policy model sampled is counted once. When the cumulative sampling number reaches the target threshold, the DRL training and sampling process is terminated. The termination condition does not depend on the complete convergence of the reward function, but only on the fact that a sufficient number of policy models have been obtained, after obtaining the original set of sub-coordinators. Then, non-blocking verification is performed on each sub-coordinator, and Algorithm 2 is used to test the sample set. Each sub-coordinator in the algorithm is verified, and valid sub-coordinators that satisfy the Trim property are selected to obtain the set of verified sub-coordinators. , Algorithm 2 is the specific verification process for the sub-coordinator: Algorithm 2: Sub-coordinator Verification Algorithm Input: Sub-coordinators obtained from DRL Controlled system ; Output: Verified sub-coordinators Or NULL, including: 2-1) Initialize the reachable state set Initialize queue ; 2-2) While queue Non-empty do; 2-3) Retrieve the current state from the queue ; 2-4) Initialize the set of executable events in the current state. ; 2-5) For each uncontrollable event do; 2-6) If Defined and Then is not defined; 2-7) Return NULL; 2-8) Else If Defined and There is a definition for then; 2-9) join in ; 2-10) End If; 2-11) End For; 2-12) For each controllable event do; 2-13) If Defined and There is a definition for then; 2-14) join in ; 2-15) End If; 2-16) End For; 2-17) If and then; 2-18) Returns NULL; 2-19) End If; 2-20) For each do; 2-21) Calculate the next state ; 2-22) If state Then was not visited; 2-23) join in and joined the team; 2-24) End If; 2-25) End For; 2-26) End While; 2-27) Initialize the common reachable state set Initialize the reverse queue ; 2-28) While queue Non-empty do; 2-29) Retrieve the state from the queue ; 2-30) For each state do; 2-31) For each event do; 2-32) If and then; 2-33) will join in Merge into the reverse queue; 2-34) End If; 2-35) End For; 2-36) End For; 2-37) End While; 2-38)If| |=| | then; 2-39) Return Verification passed; 2-40) Else; 2-41) Returns NULL, indicating a livelock exists; 2-42) End If; 4) Merging of sub-coordinators: Obtaining the set of verified sub-coordinators. Then, multiple sub-coordinators are merged, and the final coordinator is constructed by performing a union operation on the sub-coordinators: Let the sub-coordinators be... The merge coordinator is defined as follows: , where the state set , mark state set transfer function For any If it exists Make ,but The merging process of the sub-coordinators adopts Algorithm 3: Algorithm 3: Sub-coordinator merging algorithm Input: The set of verified sub-coordinators ; Output: The coordinator obtained by merging ,include: 3-1) Initialization: Let transfer function Empty; 3-2) For each sub-coordinator do; 3-3) For each state do; 3-4) If state (This state is already in the state set) (in Chinese) then; 3-5) For each event do; 3-6) If Defined and Then is not defined; 3-7) Order and order ; 3-8) If state then; 3-9) join in ; 3-10) If state then; 3-11) will join in ; 3-12) End If; 3-13) End If; 3-14) End If; 3-15) End For; 3-16) Else, state ; 3-17) will join in ; 3-18) If state then; 3-19) join in ; 3-20) End If; 3-21) End If; 3-22) End For; 3-23) End For; 3-24) Output Coordinator .

Images