Method and system for generating state feedback control strategy of delay Petri network system in non-deterministic environment

Abstract

The invention belongs to the field of discrete event system control, intelligent manufacturing and reinforcement learning, and discloses a state feedback control strategy generation method and system of a time delay Petri network system in a non-deterministic environment, and the method comprises the steps: constructing a time delay Petri network model containing controllable / uncontrollable transition and minimum transmission delay constraint, defining a time extension state formed by the identification and the enable transition residual delay; a two-stage random judgment mechanism is introduced, two types of non-deterministic factors of control execution errors and uncontrollable event preemption are modeled, and the state transition probability is deduced; a control problem is formalized into a Markov decision process, and a reward function with time consumption as a negative reward and deadlock and an ultra-long path as strong punishment is designed; and adopting a table type Q learning algorithm based on dynamic learning rate scheduling, iteratively updating an action value function in interaction with a simulation environment, and exporting a state feedback control strategy. The method can be widely deployed in discrete event system control scenes such as flexible manufacturing, logistics scheduling and multi-robot cooperation.

Description

A method and system for generating state feedback control strategies for time-delayed Petri net systems in nondeterministic environments Technical Field

[0001] This invention belongs to, but is not limited to, the fields of discrete event system control, intelligent manufacturing and reinforcement learning, and particularly relates to a method and system for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments. Background Technology

[0002] Discrete-Event Systems (DES) are a class of dynamic systems triggered by asynchronous events and characterized by discrete state transitions. They are widely found in resource-constrained and highly concurrent engineering scenarios such as discrete manufacturing, automated logistics, and robotic operations. Petri nets, due to their ability to formally express logical relationships such as concurrency, synchronization, conflict, and resource sharing, have become one of the mainstream tools for DES modeling, analysis, and control synthesis.

[0003] With the development of intelligent manufacturing and cyber-physical systems, DES control problems have the following new characteristics and technical challenges:

[0004] (1) Explicit Time Constraints: Operations such as processing, transmission and waiting have clear minimum time requirements. Traditional Petri nets are difficult to directly describe the semantics of "after enabling, at least a certain amount of time must be waited before transmission is allowed". Therefore, timed Petri nets and their variants have been widely studied.

[0005] (2) Coexistence of controllable and uncontrollable events: In actual systems, some transitions (such as equipment startup and task allocation) can be enabled or disabled by the controller, while other transitions (such as equipment failure and material arrival) are not directly controlled by the controller and their occurrence is random and preemptive.

[0006] (3) State space explosion: Concurrency and resource sharing cause the number of reachable identifiers to grow rapidly with the scale of the system. The introduction of temporal semantics further expands the dimension of state description (identifier + remaining delay), resulting in the computational complexity bottleneck of accurate analysis and strategy synthesis based on the complete temporal reachable graph.

[0007] (4) Uncertainty is ubiquitous: factors such as equipment failure, operation error, communication delay, and human intervention make the system behavior uncertain. Traditional control methods based on deterministic models will experience a significant performance decline or even failure when deviating from the assumptions.

[0008] Existing research on the above issues can be mainly divided into three categories:

[0009] (1) Model-based methods: For example, Lefebvre et al. constructed a time-extended reachability graph and formulated the minimum time reach problem as a Markov decision process, using value iteration to solve for the optimal control sequence. This method works well when the state space is manageable, but the explicit graph construction and storage limit its scalability in large-scale systems, and it is highly dependent on the accuracy of the transition probability model.

[0010] (2) Data-driven approach: In recent years, the combination of reinforcement learning and Petri nets has become a research hotspot. Hu et al. used deep reinforcement learning and graph convolutional networks to approximate the action value function; Luo et al. proposed a real-time scheduling framework based on Petri nets with place delay. However, existing methods mostly focus on deterministic environments or only consider a single type of uncertainty, lacking a unified modeling and policy learning framework for the coexistence of two types of nondeterministic factors: "control execution error" and "uncontrollable event preemption". At the same time, due to the fixed structure of neural networks, when the input state is more complex, information loss or inability to input may occur, resulting in failure to converge an effective policy.

[0011] (3) Path prediction-based methods: Traditional optimal path or optimal control sequence solution methods usually only generate a single path from the initial state to the target state. When the system deviates from the original planned path during execution due to control errors, uncontrollable events, or other random disturbances, it is often necessary to re-search for the path or plan the strategy, which makes it difficult to respond to the dynamic changes in actual operation in a timely manner.

[0012] Based on the above analysis, the urgent technical problems that need to be solved in the existing technology are:

[0013] How can we automatically learn a robust, interpretable, and non-explicit state space enumeration-free state feedback control strategy based on time-delay Petri net modeling, for non-deterministic environments with coexisting controllable / uncontrollable events, non-deterministic state transitions, and strict time constraints? Summary of the Invention

[0014] To address the problems existing in the prior art, this invention provides a method and system for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments.

[0015] This invention is implemented as follows: a method for generating a state feedback control strategy for a time-delayed Petri net system under a nondeterministic environment, characterized in that the method specifically includes:

[0016] S1: Construct a time-delay Petri net system model that includes controllable and uncontrollable transitions and assigns a minimum transmit delay constraint to each transition;

[0017] S2: Defines the time-extended state representation, which consists of the current identifier and the remaining transmit delay of each enabled instance corresponding to each enabled transition under the current identifier.

[0018] S3: Construct a two-stage stochastic decision mechanism that includes control execution errors and preemption of uncontrollable events to describe the state transition probability in a nondeterministic environment;

[0019] S4: The control problem of the time-delay Petri net system is formalized as a Markov decision process, and a reward function is designed with time consumption as negative reward and deadlock and excessively long paths as strong penalties.

[0020] S5: A tabular Q-learning algorithm based on dynamic learning rate scheduling is adopted to iteratively update the action value function during interactive sampling with the Petri net simulation environment;

[0021] S6: Derive the state feedback control strategy based on the converged Q-table to form a mapping decision table from state to control action.

[0022] Furthermore, the time-delay Petri net system consists of triples. This indicates that it includes Petri net structures. A vector that assigns the minimum unit of time to each transition. and initial identifier .

[0023] Furthermore, the time-extended state is represented as a binary tuple. ,in For identification vectors, An enable transition delay table is used; the delay table records the minimum remaining delay time for each enable instance of all enable transitions under the current identifier, and uses infinite server semantics to handle the multi-enablement case.

[0024] Furthermore, the two-stage random decision mechanism includes:

[0025] (1) Control execution error judgment, based on probability. Controllable changes in launch target; based on probability Select one transition from the candidate transition set according to a preset probability distribution as the transition to be launched;

[0026] (2) Uncontrollable preemption determination: If there are uncontrollable transitions in the current enable set, then the probability is used to determine the preemption. From the uncontrollable transitions, one is selected as the final launch transition according to a preset probability distribution; based on probability... The first-stage result is maintained; the state transition probability is determined based on the judgment results of the control execution error stage and the uncontrollable preemption stage, and the subsequent state is generated accordingly.

[0027] Furthermore, the reward function is specifically as follows:

[0028]

[0029] in This refers to the actual time consumed during this launch. The preset large penalty constant is used to distinguish between failed trajectories and normal trajectories. Step is the current step number of the ongoing event.

[0030] Furthermore, the dynamic learning rate is calculated using the following formula:

[0031]

[0032] in State-Action Cumulative number of updates since training began The decay rate is a control parameter; the dynamic learning rate is set to decrease with the cumulative number of updates of the state-action pair, so as to meet the random approximation convergence condition and improve training stability.

[0033] Furthermore, the state feedback control strategy is stored in the form of an explicit decision table, where each record is... ,in This decision table can be directly deployed on industrial controllers to achieve real-time control decisions based on the current state. The explicit decision table represents a one-to-one correspondence between explored states and preferred control actions, enabling the system to directly look up the control action based on the current state even after deviating from the original planned path due to control errors or uncontrollable events, without needing to re-plan the global path.

[0034] Another objective of this invention is to provide a state feedback control strategy generation system for a time-delayed Petri net system under a nondeterministic environment, the system specifically comprising:

[0035] Model definition module: used to receive user input of Petri net structure, transition controllability partitioning, minimum transmission delay vector, initial identifier and target identifier;

[0036] State awareness module: used to collect the current identifier and the remaining delay of each transition from the simulation environment or physical system, and construct the time-extended state;

[0037] Random event generation module: used to generate state transition samples under nondeterministic environments based on a random decision mechanism;

[0038] Reinforcement learning training module: Used to maintain and dynamically update the Q-table and the update count record table, and execute... -greedy action selection and Q-value iterative update;

[0039] The strategy export and deployment module is used to convert the Q-table into a decision policy table after training convergence and provides a standard interface for external controllers to call.

[0040] Visualization and monitoring module: used to display the reward curve, convergence status, policy table content, and state space coverage in real time during the training process.

[0041] Furthermore, the reinforcement learning training module adopts a working mode that combines offline training with online fine-tuning: the initial policy learning is completed in the digital twin environment, and the Q table can be incrementally updated according to real-time data during the actual operation phase to adapt to the slow drift of environmental parameters.

[0042] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0043] First, the unified modeling capability for nondeterministic environments: For the first time, under the time-delay Petri net control framework, a two-stage stochastic decision mechanism is used to simultaneously model two types of nondeterministic factors, namely "control execution error" and "uncontrollable event preemption", which makes up for the shortcomings of traditional methods that only consider a single source of uncertainty.

[0044] No explicit state-space enumeration required: By adopting an interactive Q-learning framework and sampling online with the Petri net simulation environment, the explicit construction and storage of large-scale time-extended reachable graphs are avoided, significantly improving scalability in large-scale systems.

[0045] Dynamic learning rate guarantees convergence: The dynamic learning rate scheduling strategy based on the number of historical updates of state-action pairs satisfies the convergence condition of random approximation, which improves the stability and convergence speed of the learning process while avoiding the burden of manual parameter tuning.

[0046] Significantly improved robustness: In nondeterministic environments, the method of this invention achieves higher cumulative rewards (i.e. lower time consumption penalties) than traditional value iteration methods in tests of all scales, demonstrating the robustness and optimization of the policy.

[0047] Policy interpretability and deployability: It adopts tabular Q-learning instead of deep neural network approximation. The generated policy is stored in the form of an explicit state-action decision table. The decision logic is transparent, which makes it easy for engineers to review, debug and modify. It does not rely on dedicated hardware such as GPUs, making it suitable for deployment in industrial controllers.

[0048] High versatility: This invention does not depend on a specific Petri net subclass and is applicable to various Petri net variants such as bounded / unbounded, ordinary / weighted, and with / without time delay; the parameters of the two-stage stochastic mechanism are adjustable, which can flexibly adapt to control systems with different confidence levels.

[0049] Secondly, as supporting evidence of the inventiveness of this invention, it is also reflected in the following important aspects:

[0050] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0051] Improve production efficiency and resource utilization: By optimizing the process to reach the target marker in the shortest possible time, production cycle time can be effectively shortened, waiting time and ineffective paths can be reduced, and the utilization efficiency of equipment, workstations and shared resources can be improved.

[0052] Upgrading from "path output" to "strategy output" enhances real-time control capabilities: Unlike traditional methods that only output a predicted optimal path or control sequence, this invention assigns a preferred action to each explored state, forming a complete state feedback control strategy. Therefore, when the system deviates from the original predetermined path due to control errors, uncontrollable events, or disturbances, it can still quickly select the corresponding action based on the current state without needing to re-predict and optimize the global path, significantly improving online response speed.

[0053] Reduced modeling and deployment costs: Compared to methods that rely on explicitly constructing a complete timed extended reachability graph and performing dynamic programming solutions, this invention can directly obtain control strategies through interactive learning, reducing the complexity and computational overhead of offline modeling for large-scale systems.

[0054] Adapting to variable-length state inputs reduces engineering implementation difficulty: Since the state of this invention is composed of "reachable identifier + residual delay of enabled transitions", the number of enabled transitions varies under different identifiers, and the state naturally has a non-fixed-length characteristic. Compared with deep reinforcement learning methods such as DQN that require fixed-dimensional input, this invention does not require additional state padding, encoding alignment or design of special network structures, thereby reducing the complexity of modeling and engineering implementation.

[0055] Enhanced adaptability and fault tolerance to uncertain industrial environments: In response to common uncontrollable events, execution errors and random disturbances in manufacturing systems, this invention can still use learned state feedback strategies to continuously guide the system to the target state, thereby improving the robustness and continuous operation capability of the system.

[0056] (2) The technical solution of this invention fills a technical gap in the industry both domestically and internationally:

[0057] This invention fills the technological gap in the shift from "optimal path planning" to "full-state feedback strategy generation": traditional methods typically focus on calculating an optimal or near-optimal path from the initial state to the target state, often requiring replanning when deviations occur during execution. This invention not only focuses on a single path but also generates corresponding optimal actions for each explored state, thus enabling immediate control decisions even when the system deviates from the original path, achieving rapid response to anomalies and disturbances.

[0058] This invention fills the gap in reinforcement learning control methods for variable-length timed extended states: addressing the problem that the number of enabled transitions in timed Petri net states varies with the identifier, resulting in non-fixed input length, this invention adopts a tabular Q-learning method instead of a deep Q-network method with fixed-dimensional input, which can complete the learning of control policies without the need to design additional complex state encoding networks.

[0059] (3) The technical solution of the present invention solves a technical problem that people have long wanted to solve but have never been able to solve successfully:

[0060] The challenge lies in generating minimum time-state feedback strategies under partially controllable and uncertain environments: Industrial settings often contain uncontrollable events, execution deviations, and random disturbances, making strategies derived offline based on ideal models difficult to apply directly. This invention transforms this problem into a Multiplication Table (MDP) approach and employs Q-learning for online iterative approximation of the optimal action value, thus solving the problems of traditional methods' strong dependence on accurate models and insufficient adaptability.

[0061] The technical challenge of timely correction after execution deviations: Traditional prediction path methods typically provide only a single path or a single action sequence. If control errors, uncontrollable transition preemption, or path deviations occur during actual execution, it often requires re-searching, predicting, or planning, resulting in long response times and difficulty meeting real-time control requirements. This invention pre-learns the optimal action for each explored state, enabling the system to quickly determine the next control action based on the current actual state even if it deviates from the original path, achieving rapid correction and continuous evolution towards the target state.

[0062] Balancing optimality, stability, and interpretability in industrial control: While deep reinforcement learning can handle complex problems, in the variable-length discrete state scenarios addressed in this invention, it often requires additional input encoding design, and training stability and interpretability are limited. This invention achieves good convergence stability and control performance in medium-scale discrete manufacturing scenarios through tabular Q-learning and adaptive learning rate design based on the number of visits, and can directly output interpretable state feedback strategies.

[0063] (3) The technical solution of the present invention overcomes technical bias:

[0064] This invention overcomes the technical bias that "reinforcement learning is difficult to directly apply to Petri net systems with temporal semantics and variable-length state structures." By defining states as "a multiset of identifiers + enable transition residual delays" and establishing a mapping relationship between actions and controllable transitions, this invention enables reinforcement learning to adapt to timed Petri net control problems.

[0065] This invention overcomes the technical bias that "deep reinforcement learning methods such as DQN are necessarily superior to tabular Q-learning methods." In the timed Petri net variable-length state scenario targeted by this invention, DQN typically requires fixed-dimensional inputs, often necessitating additional state padding, encoding, or network structure modifications. In contrast, this invention employs tabular Q-learning, which not only avoids the modeling difficulties caused by non-fixed-length inputs but also exhibits more direct and stable convergence performance in small to medium-sized discrete state spaces.

[0066] This invention overcomes the technical bias that "it is difficult to learn an effective state feedback control strategy when there are uncontrollable transitions and execution errors." By explicitly incorporating uncontrollable preemption and control error factors into the MDP transfer mechanism, this invention demonstrates that a control strategy with good performance can still be learned in uncertain environments. Attached Figure Description

[0067] Figure 1 is a flowchart of the state feedback control strategy generation method for a time-deterministic Petri net system under an undeterministic environment provided by an embodiment of the present invention;

[0068] Figure 2 is a block diagram of the state feedback control strategy generation system for a time-deterministic Petri net system under an uncertain environment provided by an embodiment of the present invention;

[0069] Figure 3 is a time-delay Petri net model of a small-scale production scheduling system provided in an embodiment of the present invention;

[0070] Figure 4 is a Petri net model of the time delay of the flexible manufacturing system provided in an embodiment of the present invention;

[0071] Figure 5 is a comparison chart of convergence time under nondeterministic conditions provided by the embodiments of the present invention;

[0072] Figure 6 is a comparison chart of cumulative rewards under nondeterministic environments provided by an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0074] As shown in Figure 1, this embodiment of the invention provides a method for generating a state feedback control strategy for a time-delayed Petri net system under a nondeterministic environment. The method specifically includes:

[0075] Step S1: Modeling the time-delay Petri net system

[0076] (1) Define Petri net structure ,in: For all limited sets in the library; For a finite set of transitions; This is the preceding correlation matrix; For post-correlation matrix; Correlation matrix .

[0077] (2) Define the transition minimum launch delay vector , Indicates change The minimum time unit that must elapse after being enabled;

[0078] (3) Divide the transition set into controllable transition sets. With uncontrollable change sets ,satisfy , ;

[0079] (4) Define the initial identifier With target identifier ;

[0080] (5) The time-delay Petri net system is represented as .

[0081] Step S2: Construction of the temporally expanded state space

[0082] (1) Define the system state as a binary tuple ,in: This is the current identifier; To enable the transition delay table, it is represented as a set:

[0083]

[0084] in For identification The enabling transition set below, Indicates change The minimum remaining latency for each currently enabled instance;

[0085] (2) Adopting infinite server semantics: when transition enables degree At that time, each enabled instance is timed independently and in parallel, and there are multiple corresponding entries in the delay table;

[0086] (3) Initial state In the middle, for all Its remaining delay is initialized as .

[0087] Step S3: Design of Nondeterministic State Transition Mechanism

[0088] (1) Define the control action space , Establish a mapping between control actions and controllable transitions. ;

[0089] (2) In the current state Next action The corresponding controllable transition of the objective is , for The following set of executable actions;

[0090] (3) Introduce a two-stage stochastic decision mechanism to simulate a nondeterministic environment:

[0091] Phase 1: Control Execution Errors

[0092] With probability choose This indicates that the execution was correct;

[0093] With probability From the set A transition is selected as a transition according to a preset probability distribution. This indicates an execution error;

[0094] Phase Two: Uncontrollable Preemption

[0095] like : based on probability The final launch change This indicates that the land has not been occupied.

[0096] With probability From the set An uncontrollable transition is selected as a subset of the inputs according to a preset probability distribution. This indicates that the property has been seized.

[0097] like Directly ordered .

[0098] In this embodiment, the preset probability distribution is a uniform distribution.

[0099] (4) Based on the above mechanism, the state transition probability function satisfy:

[0100]

[0101] in The probability of uncontrollable changes leading to failure to secure launch, when marked When uncontrollable changes are enabled, The value is 1 when there is no uncontrollable transition that enables it.

[0102] (5) Based on the final launch changes The following state transition calculation algorithm is called to generate the successor state. :

[0103] Calculate intermediate identifiers ;

[0104] Calculate the next identifier ;

[0105] right Changes in enable Update remaining delay: ,in The actual launch delay is equal to the launch transition at... The corresponding remaining delay;

[0106] right The shift towards new enabling technologies Initialize the remaining delay as ;

[0107] Combining to obtain .

[0108] Step S4: Building the Reinforcement Learning Interaction Framework

[0109] (1) Model the control problem as a Markov decision process , where the state space All reachable Action space Control actions that are mapped one-to-one with controllable transitions; state transition functions. : Defined by the two-stage stochastic mechanism in step 3; discount factor : Used to weigh long-term versus short-term returns; reward function Designed as follows:

[0110]

[0111] in: A pre-set large penalty constant is used to distinguish between failed trajectories and normal trajectories; This is the preset maximum exploration step size per round; This represents the actual time consumed in this launch; step indicates the number of state transition steps executed in the current training round.

[0112] Step S5: Tabular Q-learning based on dynamic learning rate

[0113] (1) Initialize the Q table: dimension is Initialize all unvisited state-action pairs to 0;

[0114] (2) Initialize the update count record table N: with the same dimension as the Q table, record the update count of each state-action pair, and initialize it to 0;

[0115] (3) Each round starts from the initial state Begin, execute in a loop:

[0116] use -Greedy strategy selects actions:

[0117]

[0118] Execute action c, and generate the successor state s' and immediate reward r according to the random emission mechanism of step S3;

[0119] If s' is not recorded in the Q table, then add a new row for s', and the initial Q value for all actions is 0;

[0120] Calculate the dynamic learning rate:

[0121]

[0122] in For attenuation rate control parameters, the preferred values ​​are... Furthermore, the dynamic learning rate satisfies the Robbins-Monro stochastic approximation condition: for any , , Established.

[0123] Update Q value:

[0124] renew ;

[0125] like or Or the number of steps reaches Terminate the current round; otherwise Continue iterating.

[0126] (4) Exploration rate decays at the end of each round:

[0127]

[0128] in To minimize the exploration rate, a value of 0.01 is preferred. The attenuation factor is preferably 0.99.

[0129] Step S6: Exporting the State Feedback Control Strategy

[0130] (1) After training, for each state s recorded in the Q table, select:

[0131]

[0132] (2) Output strategy table ,in This indicates that there are no actions available in a deadlock state;

[0133] (3) The strategy table can be directly deployed on the controller to realize real-time decision-making based on the current state.

[0134] As shown in Figure 2, the state feedback control strategy generation system for a time-delayed Petri net system under a nondeterministic environment provided by this embodiment of the invention specifically includes:

[0135] Model definition module: used to receive user input of Petri net structure, transition controllability partitioning, minimum transmission delay vector, initial identifier and target identifier;

[0136] State awareness module: used to collect the current identifier and the remaining delay of each transition from the simulation environment or physical system, and construct the time-extended state;

[0137] Random event generation module: used to generate state transition samples under nondeterministic environments based on a random decision mechanism;

[0138] Reinforcement learning training module: Used to maintain and dynamically update the Q-table and the update count record table, and execute... -greedy action selection and Q-value iterative update;

[0139] The strategy export and deployment module is used to convert the Q-table into a decision policy table after training convergence and provides a standard interface for external controllers to call.

[0140] Visualization and monitoring module: used to display the reward curve, convergence status, policy table content, and state space coverage in real time during the training process.

[0141] This system constructs a complete state feedback control strategy generation mechanism for time-deterministic Petri net systems under nondeterministic environments through the organic collaboration and dynamic linkage of six core modules. Each module is not an isolated unit that operates independently, but rather forms a closed-loop collaborative system through data interaction and logical connection. Its overall working principle revolves around the core link of "model definition - state awareness - stochastic simulation - reinforcement learning - policy deployment - real-time monitoring", realizing full-process controllability and traceability from system input to policy output. Its collaborative mechanism cannot be achieved through simple combination of existing technologies, as detailed below.

[0142] The model definition module, as the fundamental supporting unit of the entire system, plays a crucial role in providing a unified parameter benchmark and model specification for all subsequent modules. This module receives user input of the Petri net structure, transition controllability partitioning, minimum launch delay vector, initial identifier, and target identifier. It then standardizes and converts the various parameters to generate basic model data that the system can recognize. This data is synchronously transmitted to the state awareness module, random event generation module, and reinforcement learning training module, ensuring that each module operates based on a unified model benchmark. This avoids collaborative gaps caused by inconsistent parameters. This fundamental setting lays the foundation for the coordinated operation of the entire system and cannot be achieved by simply splicing existing model definition technologies with other modules.

[0143] The state awareness module, acting as the system's "sensing terminal," connects in real-time to the simulation environment or physical system, continuously collecting the current identifier and the remaining delays of each transition. It then fuses the collected discrete state data with the delay data to construct a time-extended state. This module is not simply a data acquisition unit; its acquisition frequency and data accuracy are dynamically adjusted by the parameters set in the model definition module. Simultaneously, it synchronizes the constructed time-extended state to the random event generation module and the reinforcement learning training module in real-time, achieving dynamic linkage between the "model baseline" and the "real-time state." This solves the technical challenge of messy state data and inaccurate model matching in non-deterministic environments. Its data fusion and dynamic synchronization mechanism is a key support for system collaboration.

[0144] The random event generation module, based on the fundamental parameters provided by the model definition module and combined with the real-time extended state transmitted by the state awareness module, generates state transition samples under nondeterministic environments through a built-in random decision mechanism. The samples generated by this module are not random or disordered, but rather strictly match the transition rules of the time-delay Petri net and the random characteristics of the nondeterministic environment. Simultaneously, the generated state transition samples are input into the reinforcement learning training module in real time, providing training data that closely reflects actual working conditions. This achieves a precise connection between "real-time state" and "random simulation." Its sample generation logic is deeply bound to model parameters and real-time states, avoiding the problem of existing random simulation techniques becoming disconnected from Petri net systems.

[0145] The reinforcement learning training module, as the "core decision-making unit" of the system, receives basic model parameters from the model definition module, real-time extended states from the state awareness module, and state transition samples from the random event generation module. It maintains and dynamically updates the Q-table and update count record table. This module executes an ε-greedy action selection strategy, iteratively updating the Q-value based on real-time states and transition samples. Its update logic is deeply correlated with the transition controllability partitioning and minimum launch delay vector depth of the model definition module. Simultaneously, it synchronizes data such as the reward curve and convergence state during training to the visualization and monitoring module. The reinforcement learning training of this module is not independent but forms a dynamic closed loop with the preceding three modules. It dynamically adjusts the training strategy based on real-time states and random samples, ensuring that the trained strategy can adapt to the dynamic changes of the nondeterministic environment. This closed-loop training mechanism cannot be achieved independently by existing reinforcement learning techniques and Petri net control techniques.

[0146] After the reinforcement learning training module converges, the policy export and deployment module receives the final Q-table transmitted from the reinforcement learning training module, converts it into a standardized decision policy table, and provides a standard interface for external controllers to call. The conversion logic of this module strictly matches the Petri net structure and transition rules defined in the model definition module, ensuring that the exported policy can be directly adapted to the target system. This achieves a seamless connection between "training policy" and "actual deployment." Its interface design and policy conversion mechanism, along with the synergy with preceding modules, avoids the incompatibility issues between existing policy export technologies and Petri net systems.

[0147] The visualization and monitoring module receives training data from the reinforcement learning training module and policy table content from the policy export and deployment module in real time. It displays the reward curve, convergence status, policy table content, and state space coverage in real time. Simultaneously, it feeds back any abnormal data detected to the model definition and reinforcement learning training modules, enabling real-time monitoring and dynamic adjustment of the entire system operation. This module's monitoring logic forms a closed loop with the data interaction between modules, ensuring that anomalies during training and deployment are detected and handled promptly, thus guaranteeing the accuracy and reliability of policy generation.

[0148] The six modules of this system form an inseparable collaborative system through data interaction, logical linkage, and dynamic closed-loop. The functionality of each module depends on the support of the others. Its overall collaborative mechanism solves the core challenge of generating state feedback control strategies for time-deterministic Petri net systems under undeterministic environments, achieving end-to-end collaborative optimization from model definition to strategy deployment. Its collaborative logic and technical implementation cannot be obtained through simple combinations of existing individual technologies, significantly enhancing the system's creativity and practicality. This invention can be applied to the following typical scenarios:

[0149] (1) Real-time scheduling of flexible manufacturing units

[0150] The state feedback strategy table generated by the method of this invention is deployed in a programmable logic controller;

[0151] In each decision cycle, the controller reads the current processing status, including the occupancy of each workstation, the buffer queue, and the remaining processing time of the equipment, and maps it to... Based on the status, a table is consulted to obtain recommended control actions, such as: starting a machine tool or transferring a workpiece to the next station;

[0152] When equipment failure (an uncontrollable event) occurs, the system automatically switches to a nondeterministic decision-making mode, but the strategy can still guide the system to a safe state.

[0153] (2) Automated warehouse inbound and outbound scheduling

[0154] Model the stacker crane, conveyor line, and warehouse entry / exit platform in the aisle as a time-delayed Petri net;

[0155] Uncontrollable changes correspond to random arrival of goods and random equipment failures;

[0156] The strategy learned by the method of this invention can dynamically determine task priorities and device allocation, maintaining high throughput under random perturbations.

[0157] (3) Multi-robot collaborative operation

[0158] Model the robot task sequence and resource consumption as a Petri net;

[0159] Control errors correspond to packet loss in communication, resulting in commands not being executed.

[0160] The robust strategy generated by the method of this invention can avoid robot deadlock and collision, and automatically degrade when communication is abnormal.

[0161] Deployment method:

[0162] Offline training: Policy learning is completed in a digital twin system or a simulation environment driven by historical data;

[0163] Online decision-making: The controller periodically executes a state perception-table lookup-action execution loop, with the cycle length matching the system's minimum time unit;

[0164] Incremental update: An optional online learning module can be added to fine-tune the Q table using actual operating data.

[0165] Example 1: Small-scale production scheduling system

[0166] (1) System modeling

[0167] This embodiment uses the Petri net model for delay in a small-scale production scheduling system, as shown in Figure 3. Its parameters are as follows: [List of locations] ; Change set Controllable transition set Uncontrollable change set Controlling the action space Mapping relationship Minimum transmit delay vector Initial identifier Target identifier .

[0168] (2) Setting of nondeterministic parameters

[0169] Control the probability of execution errors Uncontrollable preemption probability ; Reward function parameters: , , Learning parameters: , , Number of training rounds .

[0170] (3) Training process

[0171] Initialize the Q and N tables;

[0172] Each round from start;

[0173] Perform interactive sampling and Q-value update according to steps S3-S5;

[0174] The current strategy is evaluated every 100 rounds, and is fixed. Repeat for 50 rounds and take the average cumulative reward;

[0175] Record the convergence rounds and the final strategy table.

[0176] (4) Experimental results

[0177] After 1000 rounds of training, the Q-table converged, and the nondeterministic environment control strategy is shown in Table 1.

[0178] The optimal control sequence is The total expected time consumption is reduced by approximately 18% compared to the traditional value iteration method;

[0179] The policy table contains 22 states, covering all high-probability reachable states.

[0180] Table 1

[0181]

[0182] Example 2: Flexible Manufacturing System (Scale-up Experiment)

[0183] (1) System modeling

[0184] This embodiment uses the time-delay Petri net model of the flexible manufacturing system shown in Figure 4, with the following parameters: two parallel process flows and controllable transitions. Uncontrollable changes Minimum launch delay The initial and target identifiers are shown below:

[0185]

[0186] The number of products, k, varies from 3 to 16.

[0187] (2) Experimental setup

[0188] Comparison of Nondeterministic Environments: Settings , Comparison Algorithms: The control strategy solution method based on PN-Q-learning (PQ) proposed in this invention and the traditional value iteration algorithm; Hardware Environment: Intel Core i7-12700K, 32GB RAM, Python 3.9; Evaluation Metrics: Convergence time, average cumulative reward per round.

[0189] (3) Experimental results

[0190] The convergence time comparison chart and the cumulative reward comparison chart are shown in Figures 5 and 6. In the case of a small scale, the value iteration algorithm may have a certain advantage in convergence time; as the number of products k increases, starting from k=6, the convergence time advantage of the method of the present invention gradually becomes apparent, and it shows better scalability in a larger state space; in terms of cumulative reward, the method of the present invention is always superior to the value iteration algorithm, indicating that the learned strategy is better;

[0191] (4) Parameter sensitivity analysis

[0192] right and Sensitivity tests were conducted: the performance (average time and success rate) of the method of this invention fluctuated by less than ±5% within the range of 0.01 to 0.2, demonstrating good robustness.

[0193] right Conduct optimization experiments: The optimal balance between convergence speed and final performance is achieved. Too small a value (<10) leads to initial learning fluctuations. An excessively large value (>200) will result in slow convergence.

[0194] Table 2

[0195]

[0196] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0197] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for generating a state feedback control strategy for a time-delayed Petri net system under a nondeterministic environment, characterized in that, The method specifically includes: S1: Constructing a time-delay Petri net system model that includes controllable and uncontrollable transitions and assigns a minimum launch delay constraint to each transition; S2: Defining a time-extended state representation composed of the current identifier and the remaining minimum delay of all enabling degrees of each enabling transition; S3: Constructing a two-stage stochastic decision mechanism that includes control execution errors and preemption of uncontrollable events to describe the state transition probability under nondeterministic conditions; S4: Formalizing the control problem of the time-delay Petri net system as a Markov decision process and designing a reward function with time consumption as a negative reward and deadlock and excessively long paths as strong penalties; S5: Employing a tabular Q-learning algorithm based on dynamic learning rate scheduling to iteratively update the action value function during interactive sampling with the Petri net simulation environment; S6: Deriving the state feedback control strategy based on the converged Q-table to form a mapping decision table from state to control action.

2. The method for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments as described in claim 1, characterized in that, The time-delay Petri net system consists of triples. This indicates that it includes Petri net structures. A vector that assigns the minimum unit of time to each transition. and initial identifier 。 3. The method for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments as described in claim 1, characterized in that, The time-spread state is represented as a binary tuple. ,in For identification vectors, An enable transition delay table is used; the delay table records the minimum remaining delay time for each enable instance of all enable transitions under the current identifier, and uses infinite server semantics to handle the multi-enablement case.

4. The method for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments as described in claim 1, characterized in that, The two-stage random decision mechanism includes: (1) controlling the execution of error decisions, based on probability. Controllable changes in launch target; based on probability (1) Randomly select one of the other enabling transitions as the transition to be launched; (2) Uncontrollable preemption determination: if there is an uncontrollable transition in the current enabling set, then select one of the other transitions as the transition to be launched based on probability. One of the uncontrollable transitions is uniformly and randomly selected as the final launch transition; with probability... The first-stage result is maintained; the state transition probability is analytically expressed by the product of the two-stage probabilities, and the subsequent state is generated accordingly.

5. The method for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments as described in claim 1, characterized in that, The reward function is specifically as follows: ;in This refers to the actual time consumed during this launch. A negative constant that is much larger than the maximum possible path duration, where step is the current step number of the ongoing event.

6. The method for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments as described in claim 1, characterized in that, The dynamic learning rate is calculated by the following formula: ;in State-Action Cumulative number of updates since training began The decay rate is the control parameter; the learning rate sequence satisfies the Robbins-Monro stochastic approximation convergence condition.

7. The method for generating state feedback control strategies for time-delayed Petri net systems under nondeterministic environments as described in claim 1, characterized in that, The state feedback control strategy is stored in the form of an explicit decision table, where each record is... , in This decision table can be directly deployed on industrial controllers to enable real-time control decisions based on the current state.

8. A state feedback control strategy generation system based on a time-delay Petri net system under nondeterministic environment as described in claims 1-7, characterized in that, The system specifically includes: a model definition module for receiving user input of the Petri net structure, transition controllability partitioning, minimum launch delay vector, initial identifier, and target identifier; a state awareness module for acquiring the current identifier and remaining delays for each transition from the simulation environment or physical system, and constructing time-extended states; a random event generation module for generating state transition samples under nondeterministic conditions based on a random decision mechanism; and a reinforcement learning training module for maintaining and dynamically updating the Q-table and update count record table, and executing... - Greedy action selection and Q-value iterative update; Policy export and deployment module: used to convert the Q-table into a decision policy table after training convergence, and provide a standard interface for external controllers to call; Visualization and monitoring module: used to display the reward curve, convergence status, policy table content and state space coverage in real time during the training process.

9. The state feedback control strategy generation system for a time-delayed Petri net system under nondeterministic environment as described in claim 8, characterized in that, The reinforcement learning training module adopts a working mode that combines offline training and online fine-tuning: the initial policy learning is completed in the digital twin environment, and the Q table can be incrementally updated according to real-time data during the actual operation phase to adapt to the slow drift of environmental parameters.

Images