A supervised control policy generation method based on counterexample guided deep reinforcement learning

Abstract

The application discloses a kind of supervision control strategy generation methods based on counterexample guide deep reinforcement learning, comprising: one, definition model;Two, supervision control strategy generation, the method with deep Q network as reinforcement learning subject, introduce online counterexample guide verification mechanism, using counterexample search dynamic correction control permission dictionary and optimizing reinforcement learning strategy, finally coupling convergent deep network parameters and whole state space dynamic permission dictionary, generate with global non-blocking characteristics coordinated supervision control strategy.This method can meet non-blocking and approach maximum allowable supervision strategy without explicitly constructing the global state space of controlled discrete event system, balance between computational complexity and strategy allowance, provide a new technical approach for non-blocking control of super large scale discrete event system.

Description

Technical Field

[0001] This invention relates to supervised control techniques for Discrete Event Systems (DES), and more particularly to a non-blocking supervised control policy generation technique that combines directed control constraints with counterexample-guided deep reinforcement learning. Specifically, it is a supervised control policy generation method based on counterexample-guided deep reinforcement learning. Background Technology

[0002] Discrete event systems (DES) are a crucial class of dynamic systems. Their states are not continuously changing but consist of a set of independent discrete states. Furthermore, the system's state changes only due to events, meaning that state changes occur only at specific discrete time points, rather than progressing continuously over time. Over the years, the study of DES theory has evolved into a broad interdisciplinary research field, with applications such as vehicle scheduling and control in automated guided vehicle (AGV) systems. Generally speaking, rigorous mathematical modeling of DES allows for a deeper analysis of the internal operating mechanisms of complex systems, laying the foundation for improving system efficiency, reliability, and safety. In the past few decades, researchers have studied various fundamental properties of DES, such as controllability, non-blocking behavior, and maximum permissibility. However, in practical applications, simply analyzing the properties of DES is insufficient; a more critical issue is how to constrain system behavior through control mechanisms to ensure it operates according to the intended goals while meeting given specifications. To address this, researchers have proposed supervised control theory for discrete event systems. Supervised control theory provides a rigorous formal framework, typically constructing a globally reachable state space and disabling some controllable events based on the system's real-time state to constrain system behavior, ensuring that the system's closed-loop behavior always meets specifications. However, with the increasing scale of modern industry, supervised control theory faces severe challenges in practical engineering applications. The most typical challenge is state explosion, where the global state space grows exponentially with the number of components when the controlled system is composed of multiple components combined through synchronous multiplication. This makes it difficult to explicitly construct a complete system model in real-world computing environments, let alone compute the globally optimal supervisory strategy within limited memory and time constraints. To alleviate this problem, modular supervised control has been proposed, reducing computational dimensionality by designing local supervisors for each sub-specification. However, the modular approach suffers from a severe local perspective defect: even if each local supervisor is safe from a local perspective, their parallel execution is highly susceptible to instruction conflicts that can lead to system blocking. Therefore, constructing the maximum permissible and non-blocking supervisor for large-scale complex discrete event systems is a complex and challenging task.

[0003] The relevant theories of discrete event systems in existing technologies generally include: make For a set of different symbols A finite set, often called an alphabet or set, is a collection of elements. From the alphabet All strings and strings on Composition, setting Given two strings, let's assume string 1 and yes The two strings in the text, then Represented as two strings and The connection; Definition 1: In DES, a system is typically modeled as a deterministic finite state automaton, by... It means that, among them It is a finite set of states. It is a finite alphabet of events. It is a deterministic state transition function. It is the set of feasible events in a certain state, where , This is the initial state, based on the controllability of system events. It can be divided into a set of controllable events. and uncontrollable event set , , .

[0004] Definition 2: Given two deterministic finite automata (DFA): , Their synchronous product is defined as: , in It is a set of states. It is a collection of events. : It is a partial transition function, for the state and events , Definition 3: Given a system , The event set is ,in For controllable event sets and It is a set of uncontrollable events, and Regarding language If satisfied Then it is called language about and uncontrollable event set It is controllable, among which, Representation Language The prefix closure. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a supervised control policy generation method based on counterexample-guided deep reinforcement learning. This method can achieve non-blocking and near-maximum permissibility supervised policies without explicitly constructing the global state space of the controlled discrete event system, striking a balance between computational complexity and policy permissibility. It offers a novel technical approach for non-blocking control of ultra-large-scale discrete event systems.

[0006] The technical solution to achieve the objective of this invention is: A supervised control policy generation method based on counterexample-guided deep reinforcement learning includes the following steps: I. Model Definition: A discrete event system is modeled as a deterministic finite state automaton, by... It means that, among them It is a finite set of states. It is a finite set of events. It is a deterministic state transition function. It is the set of feasible events in a certain state, where ,like Then it is called It is in a deadlock state. It is the initial state. Divided into controllable event sets and uncontrollable event set , , The supervisor dynamically disables or enables in each state. Events in the system are used to constrain system behavior, and uncontrollable events are also included. Always remain executable. Supervisor. Acting on the controlled system The resulting closed-loop language is ; II. Generation of Supervision and Control Strategy: First, directed control constraints are introduced at the action level for any state in the system. Define its initial restricted action space as Apply directed control constraints to each state. To limit the system to allow at most one controllable event at any given decision moment, the actual control mode applied to the system is determined by the following formula: ,in, For action The designated unique controllable event, A set of uncontrollable events that must always be allowed. A dynamic permission dictionary, recorded in the state The system employs an expanded event set for negative example verification, with the algorithm updated online during training guided by these negative examples. The controlled system is then modeled as an MDP, using a deep Q-network to generate policies within a constrained action space. Positive rewards are given when the system reaches a marked state (e.g., all controlled units complete their assigned tasks), while larger negative rewards are applied to states that cause system blockage. This equates the problem of "constructing a non-blocking supervised policy" to "solving the optimal control problem that maximizes long-term rewards." Finally, at the end of each simulation round and before system blockage occurs, a random exploration is performed starting from the initial state, and during the exploration process… Several intermediate states are selected as backtracking states. In these states, controllable events that were originally disabled by directed control are forcibly triggered. A finite-step forward simulation is used to verify whether the subsequent system operation remains non-blocking. If the verification passes, the event is added as a "counterexample event" to the dynamic permission dictionary of the corresponding state. This allows the strategy to be gradually expanded while ensuring safety. After multiple rounds of deep reinforcement learning sampling, counterexample search and verification, and dictionary expansion, the deep Q-network parameters gradually converge. Fixing the network parameters and dictionary mapping constitutes the supervisory control strategy of this technical solution: In actual operation, for any system state... First, the optimal action is given by the deep Q-network. Then based on the optimal action Directed control constraints and dictionaries Together they constitute the control pattern of the current state: , To implement the aforementioned supervised policy generation method based on counterexample-guided deep reinforcement learning, this technical solution employs Algorithm 1. The goal of Algorithm 1 is to search for a non-blocking supervised policy in the action space under directed control constraints using deep Q-learning, given a controlled discrete event system and its local modular supervisor. Specifically, Algorithm 1 uses repeated interactive sampling with the controlled system to guide the agent to avoid control patterns that would cause blocking. The deep network parameters output by Algorithm 1, together with the dynamic permission dictionary obtained during training using Algorithm 2, constitute the final supervised control policy: that is, for any reachable global state, a set of control instructions guaranteeing non-blocking behavior can be provided. The training process of the supervised control policy is shown in Algorithm 1. Algorithm 1: Training Algorithm for Non-Blocking Coordination Policy Based on Deep Q-Learning Input: Controlled system Local modular supervisor Controllable event set Uncontrollable event set , Output: Non-blocking coordination control strategy, specifically: the final convergence depth. Network evaluation network parameters and a dynamic permission dictionary in the full state space , 1-1) Initialize depth Network evaluation network parameters and target network parameters ; 1-2) Replay the experience pool Initialize to an empty set; 1-3) All states Dynamic license dictionary Initialize to an empty set; 1-4) Total number of global training steps Initialize to 0; 1-5) Round of simulated training; 1-6) Set the current global state of the system Initialize to initial state ; 1-7) While the step size in the current round is less than the maximum limit; 1-8) Generate a random probability value that follows a uniform distribution. ; 1-9) If Current exploration rate; 1-10) Randomly select an action from the action set. ; 1-11) Else: 1-12) Use an evaluation network to select the action with the maximum Q value. ; 1-13) End If; 1-14) Construct the current state The following control modes: ; 1-15) From control mode Execute an event randomly This causes the system state to transition to the next state. ; 1-16) If the set of feasible events for the next state ; 1-17) Instant rewards Set the severe penalty value to -100; 1-18) Else: 1-19) will be rewarded instantly. Set to standard step size reward; 1-20) End If; 1-21) The four-tuple of interactive experiences Store to experience replay pool middle; 1-22) From A small batch of data is randomly sampled, and the target Q value is calculated. 1-23) Update the network parameters based on minimizing the mean squared error loss function and evaluating the deviation of the network's predicted values. ; 1-24) Determine the current state of the system. Updated to Step length Updated to ; 1-25) End While; 1-26) If the system is not blocked in this round; 1-27) Change the current state and dictionary The algorithm 2 is then used for online counterexample search and dictionary expansion. 1-28) End If; 1-29) Every fixed number of steps, the network parameters will be evaluated. Synchronize and update to the target network ; 1-30) Output the final converged evaluation network parameters. ; In the execution of Algorithm 1, steps 1-1) to 1-4) first initialize the deep reinforcement learning environment and network parameters, and then create a dynamic permission dictionary for all states. Initializing to an empty set indicates that in the early stages of training, the policy is completely constrained by strict directed control constraints. Steps 1-5) set the total number of rounds for the global simulation. Steps 1-6) Initialize the current state of the system in each round. Initial state Repeat steps 1-8) to 1-27) in a loop until the step size of the current round reaches the maximum limit, at which point the simulation for the current round stops. In steps 1-9) to 1-13), the agent uses... The strategy selects from the action space to balance exploration and exploitation; steps 1-14) are key to constructing the control pattern. Current state Control mode under It not only includes actions The specified basic events and the uncontrollable events that must be allowed It also dynamically integrates dictionaries. The stored negative examples allow the controller's decision space to dynamically expand as training progresses. In steps 1-16), if the next state... Feasible event set If the set is empty, it means that the current policy has caused the controlled system to block. In this case, the environment will impose a huge negative penalty on the agent in steps 1-17). To suppress the selection of actions that would cause system blocking, if the system remains non-blocking, then in steps 1-22) and 1-23), the algorithm retrieves data from the experience replay pool. The network parameters are updated by randomly sampling batches of data, calculating the target value, and minimizing the mean square error. This approximates the optimal state-action value function. In step 1-26), if the current simulation round ends and the system is not blocked, it indicates that the current trajectory of the system is safe. At this time, step 1-27) will be triggered, calling Algorithm 2 to actively search for and verify those events that are disabled by directed control but are actually safe. The specific execution process of Algorithm 2 is as follows: Algorithm 2: Counterexample Search and Verification Algorithm Input: the current dictionary initial state ; Output: Updated dynamic dictionary This includes every globally reachable state in the entire state space of the controlled system. The legal permission control behavior retained after verification by Algorithm 2 counterexamples 2-1) Initialize the verification event sequence It is an empty set, the current exploration step size is 0, and the current state is empty. for ; 2-2) Generate a random exploration step size ; 2-3) While the current exploration step size is less than ; 2-4) In the current state Feasible event set Randomly select an event ; 2-5) The system is in its current state Event occurred That is, the system generates deterministic state transitions. ; 2-6) Event Add to the storage verification event sequence middle; 2-7) The current state of the system Update to the next state ; 2-8) Current exploration step size ; 2-9) End While; 2-10) Record the backtracking state This is the current state. ; 2-11) Obtain the state The set of controllable events that are disabled by the current control policy is denoted as... ; 2-12) The system is in the backtracking state Force an event that is disabled to occur. This leads to deterministic state transitions. ; 2-13) The event Append to sequence In, and the current state of the system. Update to the next state ; 2-14) Generate a random concurrent verification step size Initialize the current exploration step size to 0; 2-15) While the current exploration step size is less than ; 2-16) In the current state Feasible event set Randomly select an event ; 2-17) The system is in its current state Event occurred That is, the system generates deterministic state transitions. ; 2-18) The event Add to the storage verification event sequence middle; 2-19) The current state of the system Update to the next state ; 2-20) Current exploration step size ; 2-21) End While; 2-22) Verify whether the system experiences any blockage during the operation of the complete sequence described above; 2-23) If no blockage occurs during the system's execution along this complete event sequence; 2-24) Confirm the events that were originally disabled. It is actually safe; add it as a counterexample to the backtracking state. In the dictionary, i.e., update ; 2-25) Return the updated dictionary ; 2-26) Else: 2-27) Return to the original dictionary ; In Algorithm 2, the parameter passed in from Algorithm 1 is the current dynamic dictionary. First, steps 2-1) to 2-2) initialize the verification event sequence. And the exploration step size in the first phase, from the initial state The first phase of exploration begins in steps 2-3) to 2-9). The algorithm performs random walks within the set of currently feasible events in the controlled system, aiming to find a backtracking node in the deep state space of the system. (Step 2-10) Record the current state. In steps 2-11) to 2-12), the set of controllable events that are forcibly disabled by the current control policy in this state is obtained. And randomly select a disabled controllable event. Steps 2-12) and 2-13) are the core triggering mechanism for counterexample bootstrapping, and the system is in state Force the disabled event to be executed. The state transitions to the new state. Subsequently, in the second phase of concurrent verification (steps 2-14) to 2-21), the system transitions from the new state. Continue with random exploration, step 2-22) to verify the system along the complete sequence described above (i.e., including disabled events). If the system runs along the complete sequence described above and reaches the final state without any blockages along the way, then the previously disabled sequence will be executed. In reality, this event will not cause system blockage; it is a safe "counterexample event." Therefore, in steps 2-24), the algorithm treats this safe event... Add to status Permission dictionary middle.

[0007] The non-blocking supervised control strategy is generated using a dynamic coupling update of Algorithm 1 and Algorithm 2. The overall time complexity of the supervised strategy construction is O(log n). ,in, It is the number of simulated rounds in reinforcement learning. It is the maximum number of steps in each simulation round. It is the size of the event set. It represents the number of times the negative example verification is triggered. and These represent the exploration depths of the two stages in Algorithm 2. Compared to supervised control methods, this technical solution transforms the computational burden into a sampling process linearly related to the number of simulation steps, for each decision state (at most...). (number), through neural networks in The output control mode index is completed within a time limit, while each update of Algorithm 2 is completed in a linear time limit that is proportional to the verification depth. Therefore, the overall computational cost is dominated by the number of simulation rounds and the local verification depth, effectively avoiding the state explosion problem.

[0008] Algorithm 1 employs continuous simulation and interactive sampling with the controlled object to progressively optimize the weight parameters of the deep Q-network. This allows the agent to filter action sequences that meet the non-blocking requirements from a large number of control modes. Finally, through experience replay and asynchronous updates of the target network, it ensures that the state value function converges within the constrained action space. Algorithm 2 determines the disabled events in the input. The conversion trajectory triggered in the backtracking state determines whether to modify the license dictionary. The allowed set in the algorithm, Algorithm 2, not only corrects the overly conservative nature of directed control, but also verifies it through two-stage simulation (i.e., in Algorithm 2). and This provides a broader exploratory perspective for Algorithm 1. This iterative alternation of learning and verification ensures that the supervisor's control capability remains above the logical baseline of non-blocking, and as the number of simulation rounds increases, it further enhances... The coverage of state-event pairs in the process is constantly expanding, eventually causing the generated policies to approximate the maximum permissible supervisor in terms of behavioral expression.

[0009] Compared with existing supervised control methods that rely on accurate global models or static modular combinations, the original technical solution of this application is mainly reflected in the following aspects: 1) Static initialization constraints on the action space using directed control theory: For a large-scale discrete event system composed of N synchronized local components, the size of its global state set approximately satisfies At each decision-making moment, for the set of controllable events Each event in the set can be either disabled or enabled, thus determining the size of the unconstrained action set. In such an exponentially growing action space, traditional reinforcement learning needs to blindly explore among a large number of possible combinations of controllable events. In steps 1-15 of Algorithm 1, this technical solution uses directed control theory to statically initialize the action space, restricting the system to allow at most one controllable event to occur at any decision time. In this way, the reinforcement learning agent only needs to make a decision in each state from a linear set of choosing a controllable event and not choosing a controllable event. 2) This technical solution is based on a counterexample search and verification algorithm for successful trajectory backtracking. It identifies and recovers security events that have been excessively disabled by directed control and modular methods. This technical solution uses the non-blocking trajectory obtained by Algorithm 1 as a benchmark, and then forcibly triggers the originally disabled events in the backtracking state. It then verifies whether the event is safe to update the permission dictionary. This allows the system to dynamically identify events that are deemed dangerous but are actually operable. In this way, the monitoring strategy generated by this technical solution can continuously approach the maximum permissible monitor. 3) This technical solution can gradually complete policy learning through simulation interaction without explicitly constructing the global state space. Therefore, it has better adaptability to systems that are difficult to fully and explicitly model but can provide state transition feedback.

[0010] This method can satisfy the non-blocking and approximate maximum permissibility supervision policy without explicitly constructing the global state space of the controlled discrete event system. It achieves a balance between computational complexity and policy permissibility, and provides a new technical approach for non-blocking control of ultra-large-scale discrete event systems. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the operation of the Automated Guided Vehicle (AGV) system in the embodiment. Figure 2 This is a schematic diagram of a finite state automaton consisting of 5 automated guided vehicles in the embodiment. Figure 3 This is a schematic diagram of a finite state automaton formed by the local specifications corresponding to the automated guided vehicle in the embodiment; Figure 4 This is a trend graph showing the change in the total number of legal states that the controlled system can reach in each round using different methods. Detailed Implementation

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0013] Example: In this example, the system model consists of five AGV (Automated Guided Vehicle) components and corresponding local specifications. Figure 1 As shown, the AGV travels on a fixed circular route, transporting parts to the appropriate location; Figure 2 A schematic diagram of a finite state automaton consisting of 5 automated guided vehicles; Figure 3 This is a schematic diagram of a finite state automaton formed by the local specifications corresponding to the automated guided vehicle in the embodiment; Figure 4 The experimental results show the changes in the total number of legally attainable states in the controlled system at each round for different methods; Tables 1, 2, and 3 show the backtracking states during the process of the method in this example. The available events are listed in Table 4, which shows the specific meaning of each event in this example. Table 5 shows the parameters set for the experiment in this example. The discrete event system is modeled using a finite state automaton. The system has a finite set of states. The state vector space is 7-dimensional, containing the physical location coordinates of each AGV and the logical nodes of the local supervisor. The event set is... Among them, controllable event set (e.g., AGV startup, entering the critical zone) is represented as Uncontrollable event set (e.g., AGV task completed) is indicated as This example uses a black-box system. During the interaction, data is gradually acquired, and a dynamic license dictionary is built and updated. The simulation process starts from the known initial state of the system. Starting with an initial state gain of 0, the strategy of Algorithm 1 and the counterexample-guided verification of Algorithm 2 are alternated until the parameters of the deep Q-network are basically converged and the dynamic permission dictionary is no longer updated, thus forming a complete non-blocking supervised control strategy. The implementation steps in this example include: This example sets the total number of simulation rounds to 15000, the maximum number of exploration steps per round to 200, and the initial state of the system to be known. The initial state yield is 0, corresponding to the initial dynamic license dictionary. The specific generation process of the non-blocking safety control strategy in this example is as follows: 1) First round of simulation: ① Initialize the current system state for Normalized Input a deep Q-network, with randomized network parameters. In steps 1-9) to 1-13), the agent uses a high exploration rate. A greedy strategy allows the system to acquire more information in the early stages, ② from the system Current state First, obtain the set of physically feasible events. Combined with the current action Select allowed controllable events (In this example, the startup event of AGV_1) The system executes... This leads to deterministic state transitions. Upon reaching the next state, ③ the algorithm will include a quadruple containing state transitions and immediate rewards. Store it in the experience replay pool, then update the current state to... And continue exploring, ④ assuming continuous execution until the [number]th ... During the step, the system transitions to a deeper state. At this time, the system detected This means that multiple AGVs colliding at an intersection results in a significant negative impact on the environment. The algorithm samples from the replay pool and updates the network parameters using the mean square error. This will guide the network to avoid this state in the future. Table 1: Status after the first round of simulation Available events ; 2) No. Wheel Simulation As shown in Table 2, assuming that after... Round simulations have yielded preliminary results on the non-blocking transition logic of the deep Q-network system. ① First, initialize the current system state. for Normalized Input a deep Q-network, and in steps 1-9) to 1-13), the agent adopts... Greedy strategy, ② from the system Current state First, obtain the set of physically feasible events. Combined with the current action The corresponding directed control constraint constructs the control mode, selecting the allowed controllable events. (In this example, the startup event of AGV_1) The system executes... This leads to deterministic state transitions. Upon reaching the next state, ③ the algorithm will include a quadruple containing state transitions and immediate rewards. Store it in the experience replay pool, then update the current state to... And continue exploring, ④ from status Initially, under basic control constraints, the agent continuously selects safety events and updates the system state. Eventually, the system successfully reaches the preset marked state (i.e., all AGVs have completed their tasks), and the system remains unblocked throughout the process, with the environment providing a positive reward. , at this time, the current round ends without blocking, and Algorithm 2 is officially triggered to search for and verify counterexamples: First, initialize the verification event sequence as an empty set, starting from the initial state and performing a random walk with a exploration step size of . Record the current backtracking state as . Then, the algorithm obtains the controllable events disabled by the existing policy in the state . In this example, the concurrent driving-in event is identified as disabled. Denote . Subsequently, the system forces the occurrence of this disabled event in the current state, generating a deterministic state transition . Append the event to the verification event sequence and update the current state to . Then, starting from the new state , randomly select an event from the set of feasible events after excluding the disabled events. The system continuously generates state transitions until the concurrent verification step size reaches the maximum set value . Finally, verify the end state reached by the system after running along the above event sequence. After calculation, the number of blocked branches resulting from reaching the end state from is 0. Thus, it is confirmed that the event originally conservatively disabled is actually safe. The algorithm adds it as a counterexample event to the dictionary of the state , that is, update : Table 2: State after the m-th round of simulation Available events ; 3) The n-th (1 < m < n < 15000) round of simulation: As shown in Table 3, ① First, initialize the current system state as . Input the normalized into the deep Q network. In steps 1-9) to 1-13), the agent adopts a greedy strategy. At this time, the exploration rate decays to an extremely small value, and the agent makes decisions completely using the trained evaluation network. ② Starting from the current state of the system , first obtain the set of physically feasible events . Combine the control mode constructed by the directed control constraints corresponding to the current action and select the allowed controllable event (in this example, the start event of AGV_1). The system executes This leads to deterministic state transitions. Upon reaching the next state, ③ the algorithm will include a quadruple containing state transitions and immediate rewards. Store it in the experience replay pool, then update the current state to... And continue exploring, ④ from Starting from the current state, in each step of event selection, the current state... The control mode not only includes basic directed control criteria, but also dynamically integrates the criteria accumulated over time by Algorithm 2 and written into the dynamic dictionary. Abundant counterexamples in the example (previously entered in this example) (Concurrent events, etc.) At this point, the total number of simulation rounds ends, and the algorithm outputs a converged network model and a dynamic dictionary containing safe concurrent behaviors. The generated policy strictly guarantees the non-blocking safety of the controlled system. Table 3: State after the nth round of simulation Available events ; Table 4: Specific Meaning of the Event ; Table 5 Experimental parameters .

[0014] like Figure 4 As shown, the directed control policy (green dashed line) remains at a low level (approximately 80-100 states), almost touching the horizontal axis. This verifies that directed control theory, by limiting the activation of at most one controllable event at any given time, leads to extreme conservatism in the policy. Supervised control theory combined with reinforcement learning (blue solid line) rapidly explored approximately 4000 states during the mid-training phase. However, due to the lack of an effective safety constraint mechanism, the curve exhibits violent oscillations between 4000 and 10000 rounds. This instability indicates that the general SCT+DQN algorithm alone cannot stably maintain a high permissibility policy while ensuring non-blocking behavior. In this example, the negative example guidance method (orange dashed line) shows relatively robust state growth in the early training stages due to the initialization constraints of directed control. With the intervention of the negative example guidance mechanism, the online verification module continuously identifies safety events that are incorrectly prohibited by directed control and adds them to the dynamic permission dictionary. This results in a continuous and smooth upward trend in the curve, which then converges stably to approximately 4,400 valid states after 8,000 rounds.

Claims

1. A supervised control policy generation method based on counterexample-guided deep reinforcement learning, characterized in that, Includes the following steps: I. Model Definition: A discrete event system is modeled as a deterministic finite state automaton, by... It means that, among them It is a finite set of states. It is a finite set of events. It is a deterministic state transition function. It is the set of feasible events in a certain state, where ,like Then it is called It is in a deadlock state. It is the initial state. Divided into controllable event sets and uncontrollable event set , , The supervisor dynamically disables or enables in each state. Events in the system constrain system behavior; uncontrollable events Always keep it executable, monitor Acting on the controlled system The resulting closed-loop language is ; II. Generation of Supervision and Control Strategy: First, directed control constraints are introduced at the action level for any state in the system. Define its initial restricted action space as Apply directed control constraints to each state. To limit the system to allow at most one controllable event at any given decision moment, the actual control mode applied to the system is determined by the following formula: ,in, For action The designated unique controllable event, A set of uncontrollable events that must always be allowed. A dynamic permission dictionary, recorded in the state The system employs an expanded set of events for counterexample verification, which is updated online during training guided by counterexamples. The controlled system is then modeled as an MDP, and a deep Q-network generates policies within a constrained action space. Positive rewards are given when the system reaches a marked state (e.g., all controlled units complete their assigned tasks), while larger negative rewards are applied to states that cause system blockage. The problem of "constructing a non-blocking supervision policy" is thus equivalent to "solving the optimal control problem that maximizes long-term rewards." Finally, at the end of each simulation round and before system blockage occurs, Algorithm 2 is invoked to randomly explore forward from the initial state. Several intermediate states are selected as backtracking states, and controllable events that were previously disabled by directed control are forcibly triggered in these states. A finite-step forward simulation is used to verify whether the system remains non-blocking in subsequent runs. If the verification passes, the event is added as a "counterexample event" to the dynamic permission dictionary of the corresponding state. After multiple rounds of deep reinforcement learning sampling, counterexample search and verification, and dictionary expansion, the deep Q-network parameters gradually converge. Fixing the network parameters and dictionary mapping constitutes the supervision and control policy of this invention. In actual operation, for any system state... First, the optimal action is given by the deep Q-network. Then based on the optimal action Directed control constraints and dictionaries Together, they constitute the control mode of the current state, specifically: Algorithm 1 is used to repeatedly sample and interact with the controlled system, guiding the agent to avoid control patterns that would cause blockage. The deep network parameters output by Algorithm 1, together with the dynamic permission dictionary obtained during training using Algorithm 2, constitute the final supervised control strategy: that is, for any reachable global state, a set of control instructions that guarantee non-blocking behavior can be given, including: Algorithm 1: Training Algorithm for Non-Blocking Coordination Policy Based on Deep Q-Learning Input: Controlled system Local modular supervisor Controllable event set Uncontrollable event set Output: Non-blocking coordination control strategy, specifically: the final convergence depth. Network evaluation network parameters and a dynamic permission dictionary in the full state space ,include: 1-1) Initialize depth Network evaluation network parameters and target network parameters ; 1-2) Replay Experience Pool Initialize to an empty set; 1-3) All states Dynamic license dictionary Initialize to an empty set; 1-4) Total number of global training steps Initialize to 0; 1-5) Round of simulated training; 1-6) Set the current global state of the system Initialize to initial state ; 1-7) While the step size in the current round is less than the maximum limit; 1-8) Generate a random probability value that follows a uniform distribution. ; 1-9) If Current exploration rate; 1-10) Randomly select an action from the action set. ; 1-11) Else: 1-12) Use an evaluation network to select the action with the maximum Q value. ; 1-13) End If; 1-14) Construct the current state The following control modes: ; 1-15) From control mode Execute an event randomly This causes the system state to transition to the next state. ; 1-16) If the set of feasible events for the next state ; 1-17) Instant rewards Set the severe penalty value to -100; 1-18) Else: 1-19) will be rewarded instantly. Set to standard step size reward; 1-20) End If; 1-21) The four-tuple of interactive experiences Store to experience replay pool middle; 1-22) From A small batch of data is randomly sampled, and the target Q value is calculated. 1-23) Update the network parameters based on minimizing the mean squared error loss function and evaluating the deviation of the network's predicted values. ; 1-24) Determine the current state of the system. Updated to Step length Updated to ; 1-25) End While; 1-26) If the system is not blocked in this round; 1-27) Change the current state and dictionary The algorithm 2 is then used for online counterexample search and dictionary expansion. 1-28) End If; 1-29) Every fixed number of steps, the network parameters will be evaluated. Synchronize and update to the target network ; 1-30) Output the final converged evaluation network parameters. ; The specific execution process of Algorithm 2 is as follows: Algorithm 2: Counterexample Search and Verification Algorithm Input: the current dictionary initial state ; Output: Updated dynamic dictionary This includes every globally reachable state in the entire state space of the controlled system. The legal permission control behavior retained after verification by Algorithm 2 counterexamples 2-1) Initialize the verification event sequence It is an empty set, the current exploration step size is 0, and the current state is empty. for ; 2-2) Generate a random exploration step size ; 2-3) While the current exploration step size is less than ; 2-4) In the current state Feasible event set Randomly select an event ; 2-5) The system is in its current state Event occurred That is, the system generates deterministic state transitions. ; 2-6) Event Add to the storage verification event sequence middle; 2-7) The current state of the system Update to the next state ; 2-8) Current exploration step size ; 2-9) End While; 2-10) Record the backtracking state This is the current state. ; 2-11) Obtain the state The set of controllable events that are disabled by the current control policy is denoted as... ; 2-12) The system is in the backtracking state Force an event that is disabled to occur. This leads to a deterministic state transition. ; 2-13) The event Append to sequence In, and the current state of the system. Update to the next state ; 2-14) Generate a random concurrent verification step size Initialize the current exploration step size to 0; 2-15) While the current exploration step size is less than ; 2-16) In the current state Feasible event set Randomly select an event ; 2-17) The system is in its current state Event occurred That is, the system generates deterministic state transitions. ; 2-18) The event Add to the storage verification event sequence middle; 2-19) The current state of the system Update to the next state ; 2-20) Current exploration step size ; 2-21) End While; 2-22) Verify whether the system experiences any blockages during the operation of the complete sequence described above; 2-23) If the system does not experience any blockages during the entire sequence of events; 2-24) Confirm the events that were originally disabled. It is actually safe; add it as a counterexample event to the backtracking state. In the dictionary, i.e., update ; 2-25) Return the updated dictionary ; 2-26) Else: 2-27) Return to the original dictionary .

Images