A flexible job shop scheduling method and device based on generative adversarial imitation learning

Abstract

The application provides a flexible job shop scheduling method and device based on generative adversarial imitation learning, and belongs to the field of job shop scheduling. The method comprises the following steps: obtaining a flexible job shop scheduling problem instance; constructing a flexible job shop scheduling environment; obtaining a state feature vector corresponding to a current scheduling state in the flexible job shop scheduling environment; constructing an action space; obtaining an expert state action trajectory generated by an expert scheduling strategy, and generating a policy state action trajectory by interacting the policy network with the flexible job shop scheduling environment; determining the distribution difference between the two trajectories; generating a reward value according to the distribution difference; iteratively updating the policy network according to the reward value to obtain a trained policy network; and using the trained policy network to make decisions in the scheduling environment and output a scheduling result. The method and device provided by the application can solve the problem that the reward function relies on artificial design when solving the flexible job shop scheduling by reinforcement learning.

Description

Technical Field

[0001] This application relates to the field of workshop scheduling technology, and in particular to a flexible workshop scheduling method and apparatus based on generative adversarial imitation learning. Background Technology

[0002] The flexible job shop scheduling problem is a significant extension of the traditional job shop scheduling problem. The key difference lies in the fact that each process can be processed from one of multiple available machines, breaking the machine uniqueness limitation of traditional scheduling. This means that scheduling decisions involve not only process sequencing but also machine allocation, with the two being coupled, significantly increasing the problem's complexity. This problem has wide applications in discrete manufacturing fields such as semiconductor manufacturing, automotive assembly, and aerospace manufacturing.

[0003] Currently, the main methods for solving the flexible job shop scheduling problem include exact algorithms, heuristic algorithms, and learning-based algorithms. Exact algorithms, such as branch and bound and dynamic programming, can obtain exact solutions for small-scale problems, but the solution time increases exponentially with the problem size, making them difficult to apply to real-world production scenarios. Heuristic algorithms include metaheuristic methods such as genetic algorithms and particle swarm optimization, as well as priority scheduling rules such as first-in-first-out (FIFO) and shortest processing time priority. Metaheuristic methods can obtain relatively good solutions within an acceptable time, but they lack memory of the search trajectory, requiring a re-search when the problem instance changes, resulting in weak generalization ability. Priority scheduling rules offer fast solution speeds, but their decision quality depends on the choice of the rule itself and lacks a comprehensive consideration of the global scheduling objective, typically only yielding locally optimal solutions.

[0004] In recent years, scheduling methods based on deep reinforcement learning have attracted widespread attention. These methods utilize neural networks to fit policy or value functions, enabling the agent to learn scheduling decisions through interaction with the scheduling environment. Compared to traditional methods, reinforcement learning methods offer advantages such as data-driven approaches, strong adaptability, and fast inference speed. They can quickly generate scheduling schemes after training, and are therefore considered a promising technological path for solving the flexible job shop scheduling problem.

[0005] However, existing reinforcement learning-based scheduling methods still face challenges in designing reward functions for practical applications. The reward function is a crucial signal guiding the agent's learning within the reinforcement learning framework, and its design quality directly impacts the policy's optimization direction and final performance. In existing methods, reward functions are typically designed manually, requiring designers to rely on domain experience to determine which decisions are beneficial and detrimental to optimizing the scheduling objective. This design approach is highly subjective and arbitrary. When the scheduling problem involves multiple optimization objectives, accurately defining the weights among these objectives becomes extremely difficult. Furthermore, in flexible workshop scheduling scenarios, agents typically need to progressively complete the scheduling of each process over a long time series. Only after a complete scheduling scheme is generated can global feedback on optimization objectives such as completion time be obtained. This sparse reward signal makes it difficult for the agent to reasonably attribute the final scheduling effect to each decision made in the process, resulting in low learning efficiency and slow convergence. In addition, the relationship between state-action pairs and scheduling objectives is usually non-linear, while manually designed reward functions often employ linear combinations, making it difficult to accurately express this complex non-linear relationship and limiting the upper limit of policy performance. The aforementioned problems result in existing reinforcement learning scheduling methods exhibiting drawbacks such as unstable convergence and large fluctuations in policy performance during the training process, which restricts their practical application in flexible job shop scheduling. Summary of the Invention

[0006] In view of this, this application provides a flexible job shop scheduling method and apparatus based on generative adversarial imitation learning to solve the problems of strong design subjectivity, sparse and delayed rewards, and inability to accurately describe nonlinear reward relationships caused by the reward function's reliance on manual design when solving the flexible job shop scheduling problem using reinforcement learning.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] The first aspect of this application provides a flexible shop floor scheduling method based on generative adversarial imitation learning, the method comprising:

[0009] Obtain an instance of the flexible job shop scheduling problem, which includes a task set, a machine set, and the optional processing machines and processing times for each process.

[0010] Based on the example of the flexible job shop scheduling problem, a flexible job shop scheduling environment is constructed and modeled as a Markov decision process;

[0011] In the flexible workshop scheduling environment, obtain the state feature vector corresponding to the current scheduling state;

[0012] Construct an action space, wherein the actions in the action space are formed by a combination of process selection rules and machine allocation rules;

[0013] The expert state action trajectory generated by the expert scheduling strategy is obtained, and the strategy state action trajectory is generated by interacting with the flexible workshop scheduling environment through the strategy network.

[0014] The expert state action trajectory and the policy state action trajectory are input into the discriminator network. The discriminator network determines the source of the state action pair and identifies the distribution difference between the two trajectories.

[0015] A reward value is generated based on the distribution difference, and the reward value originates from the discrimination result of the discriminator network on the state-action pair;

[0016] The policy network is iteratively updated based on the reward value until a preset training stopping condition is met, resulting in a trained policy network.

[0017] The trained policy network makes decisions in the scheduling environment, and outputs the scheduling result when the scheduling termination condition is met.

[0018] A second aspect of this application provides a flexible workshop scheduling device based on generative adversarial imitation learning, the device comprising an acquisition module, a construction module, a generation module, a determination module, an update module, and an output module;

[0019] The acquisition module is used to acquire a flexible job shop scheduling problem instance, which includes a task set, a machine set, and the optional processing machines and processing time corresponding to each process.

[0020] The construction module is used to construct a flexible job shop scheduling environment based on the example of the flexible job shop scheduling problem, and to model it as a Markov decision process.

[0021] The acquisition module is further configured to acquire the state feature vector corresponding to the current scheduling state in the flexible workshop scheduling environment;

[0022] The construction module is also used to construct an action space, wherein the actions in the action space are formed by a combination of process selection rules and machine allocation rules;

[0023] The generation module is used to obtain the expert state action trajectory generated by the expert scheduling strategy, and generate the strategy state action trajectory by interacting with the flexible workshop scheduling environment through the strategy network.

[0024] The determining module is used to input the expert state action trajectory and the policy state action trajectory into the discriminator network, and use the discriminator network to determine the source of the state action pair and determine the distribution difference between the two trajectories;

[0025] The generation module is further configured to generate a reward value based on the distribution difference, wherein the reward value originates from the discrimination result of the discriminator network on the state-action pair;

[0026] The update module is used to iteratively update the policy network according to the reward value until a preset training stopping condition is met, so as to obtain the trained policy network.

[0027] The output module is used to make decisions in the scheduling environment using the trained policy network, and outputs the scheduling result when the scheduling termination condition is met.

[0028] The flexible job shop scheduling method and apparatus based on generative adversarial imitation learning provided in this application solves the technical problems of strong design subjectivity, sparse and delayed rewards, and difficulty in characterizing nonlinear relationships caused by the reliance on manual design for reward functions in traditional reinforcement learning methods. It achieves automatic learning and optimization of scheduling strategies, and significantly improves the learning efficiency and optimization effect of scheduling strategies. Specifically, by acquiring the state feature vector corresponding to the current scheduling state, the agent is provided with a comprehensive perception of task processing progress and machine running status, enabling subsequent decisions to be made based on accurate environmental information. By constructing an action space formed by combining process selection rules and machine allocation rules, the joint decision-making of the two coupled sub-problems of machine allocation and process sequencing in FJSP is transformed into an integrated action selection at a single decision moment, effectively reducing the decision complexity of the action space. By acquiring expert state action trajectories and policy state action trajectories and inputting them into the discriminator network for source discrimination and distribution difference determination, the technical effect of learning the optimal scheduling pattern from expert examples is achieved, allowing the policy network to obtain a clear optimization direction without relying on a manually designed reward function. Based on this, reward values ​​derived from the discriminator network's discrimination results are generated according to the distribution differences. The policy network is iteratively updated based on the reward value, allowing the reward value to dynamically evolve with the training of the discriminator network and naturally capture the nonlinear relationship between state-action pairs and scheduling objectives. This overcomes the limitations of the fixed-form reward function's insufficient expressive power. At the same time, the dense reward signal effectively alleviates the reward sparsity problem caused by the long scheduling trajectory in FJSP. Furthermore, the trained policy network is obtained through iterative updates until the preset training stopping condition is met, ensuring that the policy network has stable and near-expert-level scheduling performance after training convergence. Finally, by using the trained policy network to make decisions during scheduling execution and outputting the scheduling result when the scheduling termination condition is met, end-to-end automated scheduling from problem instance input to scheduling scheme output is realized. This provides a technical means for flexible work shops to efficiently obtain high-quality scheduling schemes without the need for manual design of reward functions. Attached Figure Description

[0029] Figure 1 A flowchart illustrating an embodiment of the flexible shop floor scheduling method based on generative adversarial imitation learning provided in this application;

[0030] Figure 2 This describes the disjunction graph change process corresponding to the 3×3 FJSP instance provided in this embodiment;

[0031] Figure 3 A schematic diagram of the MDP state transition sequence considering nonlinear rewards provided for this application;

[0032] Figure 4 A schematic diagram of the strategy network structure provided in this application;

[0033] Figure 5 This is a schematic diagram of the second embodiment of the flexible workshop scheduling device based on generative adversarial imitation learning provided in this application. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0035] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0037] Example 1

[0038] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0039] Figure 1 This is a flowchart of an embodiment of the flexible workshop scheduling method based on generative adversarial imitation learning provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0040] S101. Obtain an example of a flexible workshop scheduling problem.

[0041] It's important to note that a Flexible Job Shop Scheduling Problem (FJSP) instance is a dataset obtained by digitally abstracting scheduling tasks in an actual production workshop. Specifically, it includes a set of tasks to be processed, a set of available machines in the workshop, and the optional processing machines and processing times for each process. A complete FJSP instance constitutes all the input data for a scheduling optimization problem; subsequent environment construction and optimization solutions are based on this. In terms of acquisition methods, it can be obtained by calling publicly available FJSP benchmark datasets for algorithm training and testing, or by constructing it based on order tasks, equipment ledgers, and process card data from actual production scenarios. For the latter, each customer order needs to be defined as a task, the task decomposed into several processes, and then the optional machine set and corresponding processing time for each process determined based on equipment capacity and process routes, thus forming a complete FJSP instance.

[0042] The core difference between FJSP and the traditional Job Shop Scheduling Problem (JSP) lies in the flexibility of machine selection. In JSP, each operation can only be processed on a uniquely determined machine, machine allocation is predetermined, and scheduling decisions only involve the order of operations. In FJSP, however, each operation has one or more available machines. Scheduling decisions not only need to determine the processing order of each operation but also need to select a specific machine from the set of available machines for each operation. This dual decision-making requirement increases the complexity of the problem but also provides greater flexibility for scheduling optimization.

[0043] Specifically, let the size of the scheduling problem be... ,in Indicates the number of tasks to be processed. This represents the number of machines available in the workshop. The task set can then be represented as... The set of machines can be represented as Each task Each includes several processes that satisfy a specific processing sequence. , Indicates the first The first task Each step in a task must be completed in a given sequence; that is, the next step can only begin after the previous step has been completed. Indicate process In the machine The processing time is denoted by the number of processes. If a process cannot be processed on a certain machine, the corresponding processing time is represented by the symbol "-". The number of processes in each task can vary, and the average number of available processing machines for each process is defined as the flexibility (F) of the problem instance. The higher the flexibility, the greater the flexibility of scheduling decisions, and the higher the difficulty of solving the problem.

[0044] For ease of understanding, a 3×3 (i.e., 3 tasks, 3 machines) FJSP example is given below, as shown in Table 1.

[0045] Table 1. FJSP Examples (3×3)

[0046] Please refer to Table 1 for the task. For example, it includes three processes. , , The first process Can be done in the machine (Processing time is 3) or machine Processing time is 5 seconds, but not on the machine. The next step involves further processing. Similarly, each process has its own set of available machines and corresponding processing time.

[0047] The FJSP in this application satisfies the following assumptions: (1) each task is independent of the others, and the processing time of each task on a specific machine is determined; (2) each task contains multiple operations, which must be processed in a given order, and the next operation under the same task can only start after the previous operation is completed; (3) each machine can only process one operation at any time; (4) once an operation starts processing, it cannot be interrupted until it is completed; (5) the installation time and transportation time of the operation between machines are negligible; (6) no machine failures occur, and all machines are available at any time. The above assumptions are standard settings in the field of flexible job shop scheduling, ensuring the computability of the scheduling problem and its comparability with mainstream research.

[0048] Next, in order to more concisely represent the process sequence constraints and machine allocation constraints in the FJSP scheduling process, this application further models the above-mentioned FJSP problem instance as a disjunctive graph model.

[0049] Disjunction graphs are a concept in graph theory used to describe the sequential constraints between tasks. In an FJSP disjunction graph, each task is represented as a node in the graph, the sequential constraints between tasks are represented as directed edges, and the optional processing relationships between tasks and machines are represented as undirected edges. Specifically, an FJSP disjunction graph can be represented by a quadruple. To define.

[0050] in, For a set of process nodes, each A node corresponds to a step in a task. In the diagram, it is marked with the symbol " "express. For a set of machine nodes, each A node corresponds to a processing machine, which is represented by the symbol "" in the diagram. "express.

[0051] A conjunctive edge set is used to describe the sequential constraints between different steps within the same task. In the graph, it is represented by directed edges " "Indicates" refers to two adjacent processes under the same task. and They are connected by a conjunct edge, with the direction being... point to , indicating that only when After processing, Only then can processing begin. The conjoint edge reflects the process sequence requirements within the task and is a hard constraint that must be met during scheduling.

[0052] This is the disjunctive edge set, used to describe the optional processing relationships between processes and machines. In the graph, it is represented by undirected edges. "This is indicated by the statement that any process has at least one disjunctive edge connected to a certain machine." The connection indicates that a machine can be selected for this process. As the processing machine, the edge weight represents the corresponding processing time. In Flexible Job Shop (FJSP) scheduling, each process can have multiple disjunctive edges connecting it to different machines, reflecting the core characteristic of FJSP scheduling where each process can select multiple machines. Solving the FJSP problem involves determining a unique edge from the multiple disjunctive edges to a specific machine node for each process node, and determining the start and end times of all processes.

[0053] In addition, the disjunctive graph includes two special auxiliary nodes, Start and End, representing the start and end points of the scheduling process, respectively. The Start node connects to the first step of all tasks, and the End node connects to the last step of all tasks, serving to uniformly describe the start and end of the scheduling process.

[0054] Figure 2 This embodiment illustrates the disjunction graph transformation process corresponding to the 3×3 FJSP instance. Subgraphs (a), (b), and (c) represent the initial, intermediate, and final states of the disjunction graph, respectively. In the initial state, each process node is connected to adjacent processes within the same task only via conjunctive edges and to all available processing machines via disjunctive edges; no scheduling decisions have been determined yet. As the scheduling process progresses, for each process whose processing machine and start time are determined, a corresponding directed connection is added to the graph, while the disjunctive edges between that process and other machines are removed. In the final state, each process node retains only one determined machine connection edge, and directional edges have been added between processes on the same machine, forming a complete scheduling scheme.

[0055] The FJSP problem instance obtained in the above manner is completely represented as a disjunctive graph model, which contains both process sequence constraint information and machine selection constraint information, providing a structured state representation basis for the subsequent transformation of the scheduling problem into a Markov decision process.

[0056] S102. Construct a flexible job shop scheduling environment based on the example of the flexible job shop scheduling problem, and model it as a Markov decision process.

[0057] It's important to note that the Flexible Job Shop (FJSP) scheduling problem is a type of sequential decision problem that satisfies the Markov property. The Markov property states that the state of the system at the next step depends only on the current state and the action taken, and is independent of earlier historical states. The FJSP scheduling process precisely satisfies this property: at any given scheduling moment, the subsequent available operations and machines depend only on the current processing progress of each task and the occupancy status of each machine, regardless of the specific scheduling paths before that. Therefore, FJSP can be naturally modeled as a Markov decision process and solved using reinforcement learning algorithms.

[0058] Typically, a Markov decision process can be represented by a quintuple. The composition, and the meaning of each element, are as follows:

[0059] The state space represents the set of states that the scheduling environment can be in at all possible moments. In the FJSP scheduling environment, each state... It includes the processing progress information of each task at the current moment (which processes have been completed and which processes are being processed) and the occupancy status information of each machine (which machines are idle, which machines are running and their estimated release time), corresponding to the real-time status of all process nodes and machine nodes in the aforementioned extract diagram.

[0060] This represents the action space, that is, the space in which the agent operates in each state. The set of all possible actions that can be taken. In this application, actions are used to represent scheduling decisions, including selecting the next process to be processed and assigning a specific processing machine to that process.

[0061] This represents the state transition probability, i.e., the probability of transitioning between states. Next action After that, the environment transitions to the next state. The probability of state transition is given by the process time. In FJSP, when an agent selects a process and assigns it to a machine, the processing time of that process is given by the problem instance, and the next state of the environment is uniquely determined by the completion time of that process. Since the processing time is a fixed value, the state transition is deterministic, i.e., the transition probability is 1.

[0062] Represents the reward function, i.e., in state Next action Subsequently, the environment provides the agent with an immediate reward signal. The reward function is crucial for guiding the agent to learn the optimal policy. In traditional reinforcement learning methods, the reward function needs to be designed manually, and its design quality directly affects the agent's learning performance. However, in this application, the reward function is automatically learned by the discriminator network from expert example data, thus avoiding the limitations of manual design.

[0063] This represents the discount factor, with a value range of 100%. This is used to balance the importance of current rewards and future rewards. When the value is close to 1, the agent focuses more on long-term cumulative rewards, meaning it tends to choose decisions that are beneficial to the final completion time; when... When the value is close to 0, the agent tends to prioritize immediate rewards. In this application, because the FJSP scheduling process has a long trajectory, the choice of a single step can significantly impact the final completion time of the entire scheduling scheme. Therefore, the agent needs to comprehensively consider the long-term impact of the current decision on the future state. It should be noted that the value of the discount factor γ is not fixed and can be adaptively adjusted according to the number of tasks and steps in the scheduling problem instance. In a preferred embodiment, the discount factor... Setting it to 0.99 allows the agent to optimize for long-term cumulative rewards during training.

[0064] Figure 3 A schematic diagram of the MDP state transition sequence considering nonlinear rewards provided for this application. Figure 3 As shown, the agent at each scheduling moment Observe the current state Choose an action based on the strategy. After receiving the action, the environment transitions to the next state. And provide feedback and rewards This cycle continues until a termination state is reached. During this process, the reward value... The discriminator network bases actions on the current state ( , The reward value is calculated, rather than generated by manually set rules. Because the discriminator network uses a neural network for parameterization, its output reward value can capture the complex nonlinear relationship that may exist between the state-action pair and the scheduling target, overcoming the limitation of insufficient expressive power of traditional linear reward functions.

[0065] Through the MDP modeling described above, the scheduling optimization problem of FJSP is transformed into a cumulative reward maximization problem: the agent continuously interacts with the scheduling environment through trial and error, constantly improving its strategy based on the reward signals fed back by the environment, and finally learns the optimal scheduling strategy that maximizes the cumulative reward. This transformation allows the originally complex combinatorial optimization problem to be effectively solved using reinforcement learning algorithms, laying the foundation for the introduction of the generative adversarial imitation learning framework in subsequent steps.

[0066] S103. Obtain the state feature vector corresponding to the current scheduling state in the flexible workshop scheduling environment.

[0067] It should be noted that in the constructed FJSP scheduling environment, the state feature vector corresponding to the current scheduling moment is extracted and used as the input to the subsequent policy network and discriminator network. In the reinforcement learning framework, the design of the state feature vector directly affects the agent's ability to perceive the scheduling environment and its learning efficiency. A good state feature representation should be able to comprehensively and accurately reflect the key information of the scheduling environment at the current moment, while maintaining a moderate feature dimension to avoid the curse of dimensionality.

[0068] The FJSP scheduling process involves dynamic changes in both task processing status and machine operating status. Therefore, the design of the state feature vector needs to consider both dimensions. Specifically, at each scheduling moment, the scheduling environment state is jointly determined by the completion status of each task and the processing load of each machine. The task-side state reflects the current scheduling progress, i.e., which tasks are about to complete and which have just started; the machine-side state reflects the current resource contention situation, i.e., which machines are heavily loaded and which are idle. Only by acquiring information from both aspects simultaneously can the agent make reasonable scheduling decisions: it cannot assign tasks to overloaded machines, nor can it allow machines to remain idle for extended periods.

[0069] This application extracts state features from both the task and machine dimensions to form a unified state feature vector. :

[0070] ;

[0071] in, Indicates the total number of tasks. Indicates the total number of machines. This represents the current scheduling time. Each term in this feature vector is a dimensionless numerical value, with its range normalized to [value range missing]. The interval is convenient for neural networks to process.

[0072] It should be noted that the task completion rate is used to characterize the first... The task at the current scheduling time The processing progress. In FJSP, each task consists of several operations arranged in a specific order, and the completion status of a task is measured by the proportion of its scheduled operations to the total number of operations in that task.

[0073] Specifically, first define the mask identifier. , used to indicate the first The first task The process at the current moment Has the scheduling been completed?

[0074] ;

[0075] when When, it indicates that the process has been assigned a processing machine and the processing sequence has been determined; when When the time is specified, it indicates that the process has not yet been scheduled.

[0076] Based on the aforementioned mask identifier, the task completion rate The calculation method is as follows:

[0077] ;

[0078] in, Indicates the first The total number of steps contained in a task. The value ranges from 0 to 1: a value of 0 indicates that no steps of the task have been scheduled yet, while a value of 1 indicates that all steps of the task have been scheduled. This feature can intuitively reflect the remaining workload of each task, helping the agent to identify which tasks are close to completion and which tasks are lagging behind, thereby prioritizing the scheduling of lagging tasks in the scheduling decision.

[0079] The current maximum processing time of the task is used to characterize the first... The task at the current moment The cumulative processing time of the scheduled processes reflects the total amount of processing resources invested in the task.

[0080] ;

[0081] in, Indicates the first The first task The process is carried out on the selected machine. The processing time on the surface. When When the processing time of this step is recorded in the cumulative value; when At that time, the value of this term is 0. Therefore, Essentially, it represents the sum of processing times for all scheduled operations within the task, reflecting the amount of processing work already completed. This feature is related to the task completion rate. Together, they constitute a complete description of the task processing status: This reflects how many processes have been completed. It reflects how much processing work has been completed. Combining both can more accurately depict the true progress of a task. For example, two tasks may have completed the same number of processes, but due to the different processing times for each process, their actual workload may differ significantly.

[0082] The machine's current maximum processing end time is used to characterize the first... The machine at the current moment The final completion time of all assigned processes reflects the current load level of the machine.

[0083] First, define the allocation identifier. , used to indicate the first The first task The process at the current moment Has it been assigned to the first? Table of machines:

[0084] ;

[0085] Based on the above assigned identifier, the machine's current maximum processing end time. The calculation method is as follows:

[0086] ;

[0087] in, Indicates the first The first task The first process is in the The processing end time on the machine. This value represents the maximum completion time among all assigned tasks on the machine, indicating the latest time the machine can complete all currently assigned tasks. This feature directly reflects the machine's workload. The larger the value, the more processing steps are piled up on the machine, and the later the subsequent available time window. The agent needs to consider this feature when making decisions to avoid assigning new steps to already overloaded machines, thereby achieving load balancing among machines.

[0088] Machine utilization rate is used to characterize the first The machine at the current moment The actual utilization efficiency reflects the extent to which the processing capacity of the machine is utilized.

[0089] ;

[0090] The numerator of this formula represents the first... The sum of the processing times of all assigned operations on the machine is the effective processing time that the machine has invested; the denominator is... This represents the machine's current maximum processing completion time, which is the total time span from the start of scheduling to the machine completing all assigned processes. The ratio of this to the machine utilization rate reflects the actual processing percentage of the machine during its operating time. Values ​​range from 0 to 1: a value of 0 indicates that the machine has not been assigned any processes (completely idle), and a value of 1 indicates that the machine is in an effective processing state throughout all operating hours (full load). In actual scheduling, machine utilization is an important indicator for measuring the balance of the scheduling scheme. If the utilization of each machine differs too much, it indicates that the load distribution is uneven, with some machines overloaded and others idle, which is not conducive to optimizing the overall completion time.

[0091] The four state characteristics mentioned above have a clear correspondence with the node states in the FJSP disjunction graph: task completion rate and the current maximum processing time of the task Corresponding to the scheduling status (scheduled or unscheduled) and processing time information of each process node in the extraction diagram; machine utilization rate and the machine's current maximum processing end time This corresponds to the set of assigned operations and their completion times on each machine node in the disjunctive diagram. Therefore, the state feature vector... Essentially, it transforms the graph structure information of the dissecting graph into a set of fixed-dimensional numerical features, allowing the scheduling state, originally represented by a graph structure, to be directly processed by the neural network.

[0092] All of the above state characteristic values ​​are dynamically updated as the scheduling process progresses. At each scheduling decision point... The agent reads the current state information of each task and machine from the scheduling environment and calculates the results in the manner described above. , , and Combined to form state feature vector It serves as the input to the policy network and the discriminator network.

[0093] Unlike existing technologies that use graph neural networks to extract features from scheduling states, this application uses the aforementioned state feature vectors to represent the processing progress of each task and the load level of each machine. These feature vectors do not depend on graph structure encoding and their dimensions do not change with the size of the problem instance.

[0094] S104. Construct an action space, wherein the actions in the action space are formed by a combination of process selection rules and machine allocation rules.

[0095] It's important to note that in reinforcement learning frameworks, the design of the action space directly determines the granularity and flexibility of the agent's decisions. The Functional Joints Processing (FJSP) comprises two coupled sub-problems: machine allocation and process sequencing. Machine allocation involves selecting the optimal processing machine from the available set of machines for each process; process sequencing determines the start and end times of each process while satisfying sequence constraints. These two sub-problems are interconnected. Choosing different machines for the same process will change its processing order on the machines; different orderings of processes on the same machine will also affect the machine selection space for subsequent processes. Therefore, the action space design needs to cover both decision dimensions simultaneously, enabling the agent to make dual decisions about which process to choose and which machine to use at a single decision moment.

[0096] To address this, this application designs six process selection rules and two machine allocation rules, which are combined in pairs to form 12 integrated scheduling actions, constituting the set of optional actions for the scheduling agent. At each scheduling moment, the agent selects one of these 12 integrated actions, thus simultaneously determining the process selection rule and machine allocation rule to be used at that moment. This design transforms the originally complex process-machine joint decision-making problem into a 12-classification selection problem, reducing the dimensionality of the action space while retaining decision-making flexibility.

[0097] Specifically, the six process selection rules cover different scheduling optimization perspectives: First In First Out (FIFO) selects processes according to the order in which tasks enter the scheduling system, emphasizing fairness; Shortest Processing Time (SPT) and Lowest Remaining (LRM) prioritize processes with shorter processing times at the process and task levels, respectively, emphasizing processing efficiency; Most Work Remaining (MWKR) and Most Operations Remaining (MOPNR) prioritize tasks with large remaining workloads from both processing time and number of operations dimensions, emphasizing task balance; and Flow Due Date to Most Work Remaining (FDD / MWKR) comprehensively considers the completed processing volume and the remaining processing volume of a task, emphasizing a comprehensive trade-off. Of the two machine allocation rules, the shortest processing time priority rule (Min) selects the machine with the shortest processing time, aiming to minimize the completion time of a single process; the longest processing time priority rule (Max) selects the machine with the longest processing time, which helps to achieve load balancing among machines. Among the above eight rules, different rules may produce different scheduling effects under different scheduling states. A single rule cannot achieve optimal performance in all states. Therefore, this application integrates multiple rules into the action space, allowing the agent to adaptively select the most suitable combination of rules based on the current state.

[0098] Furthermore, the above 6 process selection rules and 2 machine allocation rules are combined in pairs to form a total of 12 integrated scheduling actions. The process selection rule index is 1 to 6, and the machine allocation rule index is 1 to 2. Each combination corresponds to a unique integer action value.

[0099] The action space constructed in the above manner has the following characteristics: the action space has a fixed dimension and a moderate size, which facilitates the effective exploration and utilization of reinforcement learning algorithms; each action has a clear scheduling semantic, and the agent's decision results can be directly used for interaction with the scheduling environment; the rule combination method allows the agent to flexibly switch scheduling strategies in different scheduling states, thereby adapting to the dynamically changing scheduling environment of FJSP.

[0100] Compared with the existing technology that directly defines the action space as all possible process-machine combinations, this application constructs the action space by combining rules, fixing the action space to 12 integrated scheduling actions. The dimension of the action space does not expand with the increase of problem size, thus avoiding the curse of dimensionality. At the same time, compared with the single action space that can only select processes, the action space of this application covers both process selection and machine allocation dimensions, enabling the agent to complete the dual decision required by FJSP in a single decision moment.

[0101] S105. Obtain the expert state action trajectory generated by the expert scheduling strategy, and generate the strategy state action trajectory by interacting with the flexible workshop scheduling environment through the strategy network.

[0102] It should be noted that this application employs a generative adversarial imitation learning framework for learning the scheduling policy. The core idea of ​​this framework is to learn the optimal scheduling policy by gradually approximating the scheduling trajectory generated by the policy network with that generated by the expert scheduling policy. Therefore, the training process requires two types of data: one from the expert scheduling policy, serving as the learning target; and the other from the current policy network, serving as the object to be improved. Both types of data are sequences of state-action pairs, with each trajectory recording the entire decision-making process from the initial scheduling state to the final scheduling state.

[0103] Expert state-action trajectories are high-quality state-action pairs generated by the interaction between expert scheduling strategies and the scheduling environment. Expert scheduling strategies can be combinations of existing manually designed priority scheduling rules, or scheduling schemes pre-optimized on small-scale problem instances using exact solvers or metaheuristic algorithms.

[0104] Specifically, the process of obtaining the expert state action trajectory is as follows: First, input the current FJSP problem instance into the scheduling environment and obtain the initial state of the environment. Then, Input an expert scheduling strategy, and the expert strategy will output the optimal action corresponding to the current state. ; Scheduling environment executes actions Then, a state transition is performed to obtain the state at the next time step. Repeat the above process until the scheduling termination state is reached. In each step, the current state and the action output by the expert policy are combined to form a state-action pair. All state-action pairs at any given time are arranged in chronological order to form a complete expert state-action trajectory. In practice, the above process can be repeated for one or more FJSP problem instances to collect multiple expert state-action trajectories, forming an expert example database to improve the coverage and diversity of training data.

[0105] The policy state action trajectory is generated by the interaction between the current policy network and the scheduling environment. The policy network is a network model to be trained, and its parameters are continuously updated during the training process.

[0106] It should be noted that the output of the policy network is an integer action value, used to indicate the integrated scheduling action in the action space corresponding to the current scheduling state. Since the policy network output layer uses a Softmax function to convert the network output into an action probability distribution, the agent samples according to this probability distribution to achieve a balance between exploration and exploitation, obtaining the final integer action value. Each action in the action space is identified by an integer index, and each integer index uniquely corresponds to a combination of a process selection rule and a machine allocation rule. The integer action value output by the policy network needs to be decomposed before the specific process selection rule and machine allocation rule can be determined.

[0107] Specifically, the decomposition method is as follows: the integer action value is rounded down to obtain the process selection rule index, and the remainder is used to obtain the machine allocation rule index. Taking an action space containing 12 integrated scheduling actions as an example, let the integer action value output by the policy network be... If the operation operator is 2 (i.e., the number of machine allocation rules), then the process selection rule index is... By rounding operation The values ​​range from 1 to 6, corresponding to the following rules: First-In-First-Out (FIFO), Shortest Processing Time Priority, Longest Remaining Processing Time Priority, Lowest Ratio of Processed Time to Remaining Processing Time Priority, Highest Number of Remaining Processes Priority, and Shortest Next Processing Time Priority; Machine Allocation Rule Index By modulo operation The values ​​range from 1 to 2, corresponding to the shortest processing time priority rule and the longest processing time priority rule, respectively. Therefore, an integer action value uniquely determines a set of scheduling decision rules. For example, when the policy network outputs an action value of 7, rounding it down yields the process selection rule index 4 (corresponding to the rule with the minimum ratio of the processing time of the already processed process to the processing time of the remaining processes), and taking the remainder yields the machine allocation rule index 1 (corresponding to the shortest processing time priority rule).

[0108] In this way, the integer action values ​​output by the policy network are parsed into specific scheduling actions. These actions are then passed to the scheduling environment, which executes the action and performs a state transition, generating a new state. The policy network and the environment interact cyclically until the scheduling termination condition is met, thus obtaining a complete policy state-action trajectory.

[0109] It should be noted that in this embodiment, the policy network adopts an Actor-Critic architecture, comprising two components: an Actor network and a Critic network. The Actor network is used to output the action probability distribution, i.e., the policy, based on the current state feature vector. This probability distribution reflects the current state The Critic network is used to select the propensity for each candidate action; it is used to output a state value estimate based on the current state feature vector. This value estimate reflects the expected cumulative reward obtainable under the current policy in the current state, and is used to measure the quality of the current state. The two networks share state feature inputs but perform different functions: the Actor network is responsible for decision-making, deciding what action to take; the Critic network is responsible for evaluation, assessing the quality of the current state. Working together, the policy network can continuously optimize the quality of its scheduling decisions as it interacts with the environment.

[0110] State value estimate of Critic network output Used to guide the update direction of the Actor network. Specifically, when the agent is in state... Next action And receive a reward Then, the environment transitions to the next state. The Critic network calculates respectively and State value estimate, combined with reward Together they constitute the action Evaluation signals for quality. If the action... If the actual return is higher than expected, the Actor network increases the probability of selecting that action; otherwise, it decreases the probability of selecting that action, thereby gradually optimizing the scheduling strategy.

[0111] The policy network employs a proximal policy optimization algorithm for iterative updates. The PPO algorithm is a policy gradient algorithm based on the Actor-Critic framework. Its core idea is to limit the policy update magnitude between the new and old policies at each update, to avoid drastic fluctuations or even policy crashes due to excessively large single updates. Specifically, the PPO algorithm introduces a truncation mechanism to limit the probability ratio of the new policy to the old policy to a certain value. Within the range, The cutoff factor is set, for example, to 0.2. This mechanism ensures that the step size of each update is controlled within a reasonable range, making the training process more stable. Furthermore, the PPO algorithm supports batch updates by sampling multiple trajectories, fully utilizing environmental interaction data and exhibiting good sample efficiency, making it suitable for sequence decision problems like FJSP that require a large number of interactions.

[0112] Figure 4 The diagram below illustrates the policy network structure provided in this application, in which the Actor network and the Critic network share state feature inputs. The Actor network outputs an action probability distribution after processing by a multilayer perceptron and a Softmax layer, while the Critic network outputs a state value estimate after processing by a multilayer perceptron.

[0113] S106. Input the expert state action trajectory and the policy state action trajectory into the discriminator network, and use the discriminator network to determine the source of the state action pair and determine the distribution difference between the two trajectories.

[0114] It should be noted that the function of the discriminator network is to receive a state-action pair. As input, the output is a discrimination probability value. This probability value characterizes the probability that the input state-action pair originates from the policy state-action trajectory (i.e., generated by the current policy network) rather than the expert state-action trajectory. The optimization goal of the discriminator network is to perform accurate binary classification of the input data. For state-action pairs in the expert trajectory, the discriminator should output a value close to 0 (indicating that this is not generated by the policy network); for state-action pairs in the policy trajectory, the discriminator should output a value close to 1 (indicating that this is generated by the policy network). By continuously optimizing this classification ability, the discriminator network gradually learns the difference between the expert state-action distribution and the policy state-action distribution.

[0115] Specifically, the discriminator network employs a multilayer perceptron structure, with its input being a joint feature vector formed by concatenating the state feature vector and the action index. The state feature vector, extracted in step S103, includes the task completion rate of each task, the current maximum processing time of the task, and the machine utilization rate and the current maximum processing time of each machine. The action index is the integer action value output by the policy network in step S105. The concatenation of these two features forms the input vector of the discriminator network, which is processed through multiple fully connected layers and nonlinear activation functions, ultimately outputting a discrimination probability value by the output layer.

[0116] The training process of the discriminator network is as follows: In each training round, a batch of expert state-action pairs are randomly sampled from the expert example database. Simultaneously, the same number of policy state-action pairs are sampled from the most recently generated policy trajectories in the policy network. The two classes of samples are mixed and then fed into the discriminator network. The discriminator's output for each class of samples is calculated separately. The optimization objective of the discriminator network is to minimize the following loss function. :

[0117] ;

[0118] in, This represents the expected value regarding the strategy; This represents the state-action pairs generated by the expert strategy. This represents the state-action pairs generated by the policy network; This represents the weight parameters of the discriminator neural network. The first term of this loss function... The discriminator is encouraged to output a larger probability value (close to 1) for the policy trajectory samples, the second term The discriminator is encouraged to output smaller probability values ​​(close to 0) for expert trajectory samples. These two factors work together to enable the discriminator network to gradually learn to distinguish between the two types of trajectories. The parameters of the discriminator network... The gradient descent method is used for updating, and the update method is as follows: ,in is the learning rate of the discriminator network.

[0119] After the discriminator network completes its parameter updates, its output can be used to calculate the reward value. Specifically, the state-action pairs generated during the interaction between the policy network and the environment are used. Input the discriminator network to obtain the discrimination probability value. This probability value reflects the degree of matching between the current state-action pair and the expert state-action distribution. The closer the probability value is to 0, the greater the difference between the state-action pair and the expert distribution; the closer the probability value is to 1, the closer the two are. Further processing of this probability value in subsequent steps generates a reward signal to guide policy network updates.

[0120] During training, the discriminator network and the policy network form an adversarial relationship. The policy network continuously updates itself, attempting to generate state-action pairs that increasingly resemble the expert's trajectory to deceive the discriminator network. The discriminator network, in turn, continuously updates itself, striving to more accurately distinguish between the two types of trajectories. Through this game-like interaction, they co-evolve. The data distribution generated by the policy network gradually approximates the expert's data distribution, and the discriminator network's ability to distinguish between the two types of data continuously improves. When training tends to converge, the state-action distribution generated by the policy network becomes consistent with the expert's, and the discriminator network can no longer effectively distinguish the origins of the two types of trajectories. At this point, the policy network achieves scheduling performance comparable to or even better than the expert scheduling strategy.

[0121] S107. Generate a reward value based on the distribution difference, wherein the reward value originates from the discrimination result of the discriminator network on the state-action pair.

[0122] In traditional reinforcement learning methods, reward functions need to be designed manually. For complex scheduling problems like FJSP, manually designing reward functions faces several difficulties. First, designers need to rely on experience to judge which state-action pairs are advantageous and which are disadvantageous, which is subjective and makes it difficult to guarantee optimality. Second, the optimization objective of FJSP is to minimize the maximum completion time, but it is difficult to assess the impact of the current decision on the final goal before scheduling is completed, resulting in sparse and delayed reward signals. In addition, there are often complex nonlinear relationships between state-action pairs and the final scheduling goal, and linear combination reward functions cannot accurately characterize these relationships. These difficulties make the design of reward functions in traditional methods a bottleneck restricting the performance of reinforcement learning scheduling.

[0123] This application effectively avoids the aforementioned problems by automatically learning the reward function through a discriminator network. During training, the discriminator network gradually learns the difference between the expert's state-action distribution and the policy's state-action distribution. This difference itself can serve as a basis for evaluating the quality of the current policy. The closer the distribution of the state-action pairs generated by the policy network is to the expert's trajectory, the closer the current policy is to the expert level, and therefore, it should receive a higher reward. The discriminant probability value output by the discriminator network is a quantitative representation of this closeness, and can thus be directly mapped to a reward signal.

[0124] Specifically, at each scheduling moment The current state action generated by the interaction between the policy network and the environment The input is a discriminator network, which outputs a discrimination probability value after forward propagation. This discriminant probability value represents the probability that the current state action pair originates from the expert state action trajectory, and its value ranges from 0 to 1. The discriminator network continuously optimizes its discriminative ability during training, enabling this probability value to accurately reflect the degree of matching between the current state action pair and the expert distribution.

[0125] It should also be noted that when the state-action pairs generated by the policy network are difficult to distinguish from expert trajectories, it indicates that the current policy has approached expert level. In the discriminator network's output, this manifests as a discrimination probability value close to a preset reference value. For example, using a preset reference value of 0.5 as the critical point, when the discrimination probability value is close to 0.5, it means the discriminator network cannot effectively distinguish whether the input data comes from expert trajectories or policy trajectories. At this point, the two sets of data have a high degree of overlap in distribution, and the policy network has well imitated the expert scheduling policy, thus deserving a higher reward. When the discrimination probability value is far from 0.5, it means the discriminator network can clearly distinguish between the two types of trajectories, indicating that the data generated by the policy network still has a significant gap from the expert distribution, thus deserving a lower reward.

[0126] Based on the above principles, this application determines the corresponding reward value according to the difference between the discrimination probability value and the preset reference value; the smaller the difference between the two, the larger the corresponding reward value. In a preferred embodiment, the reward value... Calculate as follows:

[0127] ;

[0128] in, For the discriminator network in state With action The output probability value is given by the formula. This formula means that when the discrimination probability value is... When it equals 0.5, At this time, the reward value The value tending towards positive infinity indicates the maximum reward; when When approaching 0 or approaching 1, The reward value approaches 0.5. The difference is relatively small. This mapping relationship ensures the design principle that the smaller the difference between the discrimination probability value and the preset reference value, the larger the reward value. It should be noted that the above formula is only a preferred example. In practical applications, any reward function design that follows the mapping principle of the smaller the difference between the discrimination probability value and the preset reference value, the larger the reward value is acceptable.

[0129] In this way, the output of the discriminator network is converted into a dense reward signal at each scheduling time. In this system, agents receive a reward value immediately after performing an action, rather than receiving a single feedback at the end of the scheduling process. This dense reward characteristic effectively overcomes the reward sparsity problem in FJSP, enabling the policy network to obtain meaningful feedback signals at every step of the scheduling process, thereby accelerating convergence and improving training stability.

[0130] Optionally, in this embodiment, the discriminator network includes an input layer, at least one hidden layer, and an output layer. The input layer receives state-action pairs. The input is a joint feature vector formed by concatenating the state feature vector and the action index. At least one hidden layer employs a fully connected structure to perform nonlinear transformations and feature extraction on the input vector. The number of nodes and layers in the hidden layer can be adjusted according to the problem size. The output layer outputs the discrimination probability value corresponding to the input state-action pair. This probability value represents the probability that the input state action pair originates from the expert state action trajectory. The output layer is connected to a probability mapping activation function layer, which maps the discrimination probability value to a preset probability interval. Preferably, the probability mapping activation function layer uses the Sigmoid activation function to map the original output of the output layer to the (0, 1) interval, obtaining the final discrimination probability value. The specific network structure parameters of the discriminator network can be: the discriminator network contains one hidden layer with 128 nodes, using the Tanh function as the activation function; the output layer uses the Sigmoid activation function.

[0131] The reward value generated in this step will serve as the direct basis for updating the policy network in subsequent steps. The closer the discrimination probability value is to the preset reference value, the larger the reward value, and the more the policy network tends to reinforce the decision path that produces the state-action pair during updates. Conversely, the farther the discrimination probability value is from the preset reference value, the smaller the reward value, and the more the policy network tends to avoid the decision path that produces the state-action pair during updates. Thus, the distribution difference output by the discriminator network is successfully transformed into a reward signal usable by the policy network, achieving the technical effect of automatically learning the reward function from expert examples without manual design.

[0132] S108. Iteratively update the policy network according to the reward value until the preset training stopping condition is met, and obtain the trained policy network.

[0133] As described above, the policy network adopts an Actor-Critic architecture, where the Actor network outputs the action probability distribution based on the state feature vector, and the Critic network outputs the state value estimate based on the state feature vector. The policy network is iteratively updated using a near-end policy optimization algorithm, and its parameter updates depend on the reward value output by the discriminator network in step S107. .

[0134] In each iteration, the current policy network parameters are first fixed, allowing it to interact with the FJSP scheduling environment to generate several complete policy state-action trajectories. Each time step in the trajectory includes the state, action, and reward value calculated by the discriminator network. A complete trajectory is defined as {s0, a0, r0, s1, a1, r1, ..., s...}. T The entire decision-making process, from the initial state to the final state, and the corresponding real-time feedback, were recorded. Subsequently, the advantage function was calculated using the collected trajectory data. Advantage function Used to measure the state Select action The comparison of the merits of following the current strategy's average level is calculated as follows:

[0135] ;

[0136] in, As a discount factor, The state value estimate output by the Critic network. This is the reward value output by the discriminator network. A positive advantage function indicates that the action is better than the average level of the current policy, and its probability of being selected should be increased; a negative advantage function indicates that the action is worse than the average level, and its probability of being selected should be decreased.

[0137] Using the advantage function, the loss function of the Actor network can be further calculated:

[0138] ;

[0139] in, This indicates the difference between the new strategy and the old strategy in the state. Select action The probability ratio, i.e. Unlike the reward value output by the discriminator network. ; The clip function is the cutoff factor; it limits the probability ratio to a certain value. Within a certain range, to prevent excessively large update fluctuations.

[0140] The loss function of the Critic network is expressed as the mean square error of the time difference error:

[0141] ;

[0142] This loss function enables the Critic network to learn accurate state value estimates step by step, thereby providing a reliable advantage function estimate for the Actor network.

[0143] In a preferred embodiment, the hyperparameters of the policy network are set as follows: the learning rate of the Actor network is... The learning rate of the Critic network is , cutoff factor Discount factor The above parameters can be adjusted according to the size and complexity of the specific problem instance.

[0144] The policy network and discriminator network are iteratively updated using an alternating training method. In each round, the policy network parameters are fixed, and the discriminator network parameters are updated using the loss function of the discriminator network; subsequently, the discriminator network parameters are fixed again, and the policy network parameters are updated using the reward value output by the discriminator network through the PPO algorithm. The policy network and discriminator network thus alternately optimize and compete with each other, and the scheduling trajectory generated by the policy network gradually approximates the distribution of the expert trajectory.

[0145] When the number of training rounds reaches a preset value, the training is deemed to have met the stopping condition. The parameters of the current policy network are saved, resulting in the trained policy network. In this embodiment, the number of training rounds is set to 1500. Through the above method, the policy network, under the guidance of continuous interaction with the scheduling environment and the reward of the discriminator network, gradually optimizes its scheduling decision strategy, eventually converging to a policy network model that can generate a near-expert-level scheduling scheme.

[0146] S109. The trained policy network is used to make decisions in the scheduling environment. When the scheduling termination condition is met, the scheduling result is output.

[0147] It should be noted that in step S108, the policy network, through training with the generative adversarial imitation learning framework, has had its parameters tuned to generate near-expert-level scheduling policies. After training, the policy network no longer needs to interact with the discriminator network but is used independently for actual scheduling decisions. Specifically, for a given FJSP problem instance, the scheduling environment starts from the initial state... Initially, the trained policy network at each scheduling time... Receive current state The corresponding state feature vector is used as input, and after forward propagation through the Actor network, an action probability distribution is output. Integer action values ​​are sampled from this distribution. Subsequently, these integer action values ​​are analyzed to determine the process selection rules and machine allocation rules for the current moment, and the process to be processed and its corresponding processing machine are selected according to the rule combination. After the scheduling environment executes this scheduling action, the state of each task and machine is updated, and the process enters the next moment. This process continues until the scheduling termination condition is met.

[0148] In the actual scheduling process, the action value output by the policy network is processed according to the encoding-decoding method described in step S105. Specifically, the policy network outputs an integer action value, which is used to obtain the process selection rule index through rounding and the machine allocation rule index through modulo operation. The process selection rule index is mapped to one of the six process selection rules, and the machine allocation rule index is mapped to one of the two machine allocation rules. The combination of the two determines the complete scheduling decision at the current moment.

[0149] It should also be noted that the scheduling termination condition is determined using a mask matrix. This application models the scheduling completion status of tasks through a mask mechanism. Specifically, a two-dimensional mask matrix is ​​established with dimensions corresponding to the number of tasks and the maximum number of processes. Let the number of tasks be... The maximum number of steps in all tasks is Then the mask matrix The size is Each element in the matrix Corresponding to the first The first task The scheduling completion status of each process, among which , .when When, it indicates that the process has not yet been scheduled; when When the mask matrix is ​​set to 0, it indicates that the scheduling of that process has been completed. Initially, all elements of the mask matrix are 1, meaning that no processes have been scheduled. At each scheduling decision point, when a process is selected and assigned a processing machine, the corresponding mask matrix element is set to 0. When all elements of the mask matrix are 0, it indicates that all processes of all tasks have been scheduled, and the scheduling termination state has been reached.

[0150] Based on the masking mechanism described above, the termination decision of the scheduling process is transformed into a clear condition: When this condition is met, the scheduling process terminates, and the processing machines and processing sequence for all processes are determined.

[0151] When the scheduling termination condition is met, the scheduling environment summarizes all scheduling decision records and outputs a complete scheduling result. The scheduling result includes: the start and end times of each process in each task, the processing machine corresponding to each process, the processing sequence of processes on each machine, and the maximum completion time. Maximum completion time It equals the maximum value among the processing times of all processes, that is This is used to evaluate the optimization effect of the final scheduling scheme. In practical applications, the scheduling results are usually presented to the scheduler in the form of a Gantt chart so that they can intuitively view the time arrangement of each process on each machine, assisting the manufacturing execution system in production scheduling.

[0152] The method provided in this embodiment introduces a generative adversarial imitation learning framework. It utilizes a discriminator network to automatically learn the state-action distribution from expert state-action trajectories and maps the discrimination results to reward values. This eliminates the need for manual design of reward functions, avoiding policy bias and training instability problems caused by the strong subjectivity of reward function design, sparse and delayed rewards, and difficulty in characterizing nonlinear relationships in traditional reinforcement learning methods. Furthermore, because the discriminator network uses a neural network for parameterized representation, its output reward value can naturally capture the nonlinear relationship between state-action pairs and the scheduling target, overcoming the limitations of insufficient expressive power of linear reward functions. In terms of the training mechanism, the policy network and discriminator network alternately optimize in a game-like process. This ensures that the policy network consistently imitates a specific expert trajectory in a way that the discriminator cannot distinguish as the target, rather than statically mimicking it. This not only allows the scheduling performance of the policy network to reach a level comparable to that of experts after training convergence but also holds promise for discovering scheduling schemes that surpass expert policies in specific scheduling scenarios. Furthermore, this embodiment addresses the characteristic of FJSP, which requires simultaneously solving two coupled sub-problems: machine allocation and process sequencing. It designs a four-dimensional state feature vector encompassing task completion rate, current maximum processing time of the task, machine utilization rate, and current maximum processing time of the machine. It also constructs a 12-fold integrated scheduling action space, formed by combining six process selection rules and two machine allocation rules. This allows the agent to simultaneously make dual decisions—which process to select and which machine to use—at a single decision moment, effectively avoiding suboptimal solutions caused by treating the two sub-problems separately. Simultaneously, this embodiment uses a two-dimensional mask matrix to model the scheduling completion state of each task and process, transforming the scheduling termination condition into a mathematical condition of whether all matrix elements are zero, thus achieving precise determination of the scheduling termination condition.

[0153] Example 2

[0154] Corresponding to the aforementioned embodiment of a flexible shop floor scheduling method based on generative adversarial imitation learning, this application also provides an embodiment of a flexible shop floor scheduling device based on generative adversarial imitation learning.

[0155] Figure 5 This is a schematic diagram of the second embodiment of the flexible workshop scheduling device based on generative adversarial imitation learning provided in this application. Please refer to... Figure 5 The apparatus provided in this embodiment includes an acquisition module 510, a construction module 520, a generation module 530, a determination module 540, an update module 550, and an output module 560.

[0156] The acquisition module 510 is used to acquire a flexible job shop scheduling problem instance, which includes a task set, a machine set, and the optional processing machines and processing time corresponding to each process.

[0157] The construction module 520 is used to construct a flexible job shop scheduling environment based on the example of the flexible job shop scheduling problem, and model it as a Markov decision process.

[0158] The acquisition module 510 is also used to acquire the state feature vector corresponding to the current scheduling state in the flexible workshop scheduling environment;

[0159] The construction module 520 is also used to construct an action space, wherein the actions in the action space are formed by a combination of process selection rules and machine allocation rules;

[0160] The generation module 530 is used to acquire the expert state action trajectory generated by the expert scheduling strategy, and to generate the strategy state action trajectory by interacting with the flexible workshop scheduling environment through the strategy network.

[0161] The determining module 540 is used to input the expert state action trajectory and the policy state action trajectory into the discriminator network, and use the discriminator network to determine the source of the state action pair and determine the distribution difference between the two trajectories.

[0162] The generation module 530 is further configured to generate a reward value based on the distribution difference, wherein the reward value originates from the discrimination result of the discriminator network on the state-action pair;

[0163] The update module 550 is used to iteratively update the policy network according to the reward value until a preset training stopping condition is met, so as to obtain the trained policy network.

[0164] The output module 560 is used to make decisions in the scheduling environment using the trained policy network, and output the scheduling result when the scheduling termination condition is met.

[0165] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0166] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0167] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0168] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A flexible shop floor scheduling method based on generative adversarial imitation learning, characterized in that, The method includes: Obtain an instance of the flexible job shop scheduling problem, which includes a task set, a machine set, and the optional processing machines and processing times for each process. Based on the example of the flexible job shop scheduling problem, a flexible job shop scheduling environment is constructed and modeled as a Markov decision process; In the flexible workshop scheduling environment, obtain the state feature vector corresponding to the current scheduling state; Construct an action space, wherein the actions in the action space are formed by a combination of process selection rules and machine allocation rules; The expert state action trajectory generated by the expert scheduling strategy is obtained, and the strategy state action trajectory is generated by interacting with the flexible workshop scheduling environment through the strategy network. The expert state action trajectory and the policy state action trajectory are input into the discriminator network. The discriminator network determines the source of the state action pair and identifies the distribution difference between the two trajectories. A reward value is generated based on the distribution difference, and the reward value originates from the discrimination result of the discriminator network on the state-action pair; The policy network is iteratively updated based on the reward value until a preset training stopping condition is met, resulting in a trained policy network. The trained policy network makes decisions in the scheduling environment, and outputs the scheduling result when the scheduling termination condition is met.

2. The method according to claim 1, characterized in that, The state feature vector includes the task completion rate and the current maximum processing end time of each task, as well as the machine utilization rate and the current maximum processing end time of each machine. The task completion rate is obtained by the ratio of the number of completed scheduled processes to the total number of task processes. The current maximum processing end time of the task is obtained by taking the maximum completion time of the already completed scheduled processes; The machine utilization rate is obtained by the ratio of the machine's cumulative processing time to the machine's current maximum processing end time; The maximum processing time of the machine is obtained by the maximum completion time of the process assigned to the machine.

3. The method according to claim 1, characterized in that, The process selection rules include at least one of the following rules: First-in, first-out rule; Shortest processing time priority rule; The rule of prioritizing the longest remaining processing time; The rule of minimum ratio of processing time of completed process to processing time of remaining process is applied. The rule that prioritizes the process with the most remaining steps; The rule of prioritizing the process with the shortest processing time for the next step.

4. The method according to claim 1, characterized in that, The machine allocation rules include at least one of the following rules: Shortest processing time priority rule; The longest processing time takes precedence.

5. The method according to claim 1, characterized in that, The policy network outputs an integer action value to indicate the corresponding action in the action space; each action in the action space is identified by an integer index, and the process selection rule index is obtained from the integer action value by rounding down, and the machine allocation rule index is obtained from the integer action value by taking the remainder of the number of machine allocation rules.

6. The method according to claim 1, characterized in that, The scheduling termination condition is determined using a mask matrix; Construct a two-dimensional mask matrix with dimensions corresponding to the number of tasks and the maximum number of processes. Each element in the mask matrix is ​​used to represent the scheduling completion status of the corresponding process. When all matrix elements in the mask matrix represent that the corresponding process has been scheduled, it is determined that the scheduling has been terminated.

7. The method according to claim 1, characterized in that, The step of generating a reward value based on the distribution difference includes: Obtain the discrimination probability value output by the discriminator network for the current state-action pair, where the discrimination probability value represents the probability that the state-action pair originates from the expert state-action trajectory; The corresponding reward value is determined based on the difference between the discrimination probability value and the preset reference value. The smaller the difference between the discrimination probability value and the preset reference value, the larger the corresponding reward value.

8. The method according to claim 7, characterized in that, The discriminator network includes an input layer, at least one hidden layer, and an output layer; The input layer receives state action pairs as input; The output layer outputs the discrimination probability value corresponding to the state-action pair; The output layer is connected to a probability mapping activation function layer, which maps the discrimination probability value to a preset probability interval.

9. The method according to claim 1, characterized in that, The policy network includes an Actor network and a Critic network; The Actor network takes the state feature vector as input and outputs an action probability distribution; The Critic network takes the state feature vector as input and outputs the state value. The policy network is iteratively updated using a near-end policy optimization algorithm, and the update range between the new and old policies is limited during the iterative update process.

10. A flexible workshop scheduling device based on generative adversarial imitation learning, characterized in that, The device includes an acquisition module, a construction module, a generation module, a determination module, an update module, and an output module; The acquisition module is used to acquire a flexible job shop scheduling problem instance, which includes a task set, a machine set, and the optional processing machines and processing time corresponding to each process. The construction module is used to construct a flexible job shop scheduling environment based on the example of the flexible job shop scheduling problem, and to model it as a Markov decision process. The acquisition module is further configured to acquire the state feature vector corresponding to the current scheduling state in the flexible workshop scheduling environment; The construction module is also used to construct an action space, wherein the actions in the action space are formed by a combination of process selection rules and machine allocation rules; The generation module is used to obtain the expert state action trajectory generated by the expert scheduling strategy, and generate the strategy state action trajectory by interacting with the flexible workshop scheduling environment through the strategy network. The determining module is used to input the expert state action trajectory and the policy state action trajectory into the discriminator network, and use the discriminator network to determine the source of the state action pair and determine the distribution difference between the two trajectories; The generation module is further configured to generate a reward value based on the distribution difference, wherein the reward value originates from the discrimination result of the discriminator network on the state-action pair; The update module is used to iteratively update the policy network according to the reward value until a preset training stopping condition is met, so as to obtain the trained policy network. The output module is used to make decisions in the scheduling environment using the trained policy network, and outputs the scheduling result when the scheduling termination condition is met.

Images