Flexible job shop scheduling method, electronic device, medium and product

Abstract

Embodiments of the present disclosure disclose a flexible job shop scheduling method, device, medium and product. The method comprises: obtaining current state information of a work flow, wherein a dependency relationship between processes is uniformly described by a directed acyclic graph; determining a set of currently executable processes based on the current state information and the directed acyclic graph; inputting the current state information and the set of executable processes into a reinforcement learning model to decide a process to be executed and assign a machine to the process; and executing the process. Embodiments of the present disclosure can accurately model complex dependency relationships between multiple processes of different jobs, generate optimized process scheduling decisions, adapt to dynamic job arrivals, and fully utilize the parallel processing capabilities of machines.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to a flexible workshop scheduling method, electronic equipment, storage medium, and program products. Background Technology

[0002] The Flexible Job Shop Scheduling Problem (FJSP) is a classic problem in manufacturing system scheduling. In FJSP, there are multiple jobs and multiple machines. Each job consists of several operations, and each operation can be processed from one of several available machines. Traditional FJSP typically assumes that jobs are processed in a fixed order and that all jobs are known at the start of scheduling. However, in actual manufacturing processes, complex dependencies often exist between jobs. For example, in automobile manufacturing, different components such as the body, doors, engine, and seats each have independent processes, which are produced in parallel and eventually converge on the final assembly line, forming a complex dependency structure with multiple sources, branches, and convergences. The linear sequence constraints of traditional FJSP are insufficient to effectively describe such scenarios.

[0003] In existing technologies, some studies have introduced Directed Acyclic Graphs (DAGs) to describe complex dependencies between jobs. However, existing FJSP solution methods under DAG constraints still have the following shortcomings: First, most methods assume that all jobs are known and fixed at the start of scheduling, making it difficult to handle dynamic scenarios where jobs continuously arrive; second, the solution quality is limited, and heuristic methods usually only provide feasible solutions rather than optimal solutions.

[0004] Therefore, a flexible job shop scheduling method is needed that can support complex dependencies described by DAG and adapt to dynamic operating environments. Summary of the Invention

[0005] This invention provides a flexible job shop scheduling method, electronic equipment, storage medium, and program product.

[0006] A flexible job shop scheduling method includes:

[0007] Obtain the current status information of the workflow, which includes multiple jobs and multiple machines. The dependencies between the procedures of the multiple jobs are uniformly described by DAG. Each job contains a subset of procedures in the DAG. Each procedure corresponds to at least one optional machine. Each optional machine is selected from the multiple machines.

[0008] Based on the current state information and the DAG, determine the set of currently executable processing steps;

[0009] The current state information and the set of processes to be processed are input into a reinforcement learning model, so that the reinforcement learning model decides which process to be executed in the current process from the set of processes to be executed, and allocates a machine to the process from at least one selectable machine corresponding to the process to be executed; and

[0010] The process to be performed is executed on the assigned machine.

[0011] In one embodiment, the reinforcement learning model is further included in a training process; wherein, during the training process, for each training sample job, when it is detected that all the steps contained in the training sample job have been completed, a reward value is calculated based on the completion information of the training sample job, and the parameters of the reinforcement learning model are updated based on the reward value.

[0012] In one embodiment, calculating the reward value based on the completion information of the training sample job includes: obtaining the number of processes contained in the process subset of the training sample job in which all processes have been completed; obtaining the actual processing time of the training sample job from the start of the first process to the completion of the last process; and calculating the reward value based on the number of processes and the actual processing time.

[0013] In one embodiment, the training process further includes: obtaining the deadline for training sample jobs in which all procedures have been completed; adjusting the reward value based on the deadline, wherein completing ahead of time results in a positive reward, completing on time results in a zero reward, and completing late results in a negative reward.

[0014] In one embodiment, calculating the reward value based on the number of processes and the actual processing time includes: for training sample jobs with the same number of processes, the shorter the actual processing time, the higher the reward.

[0015] In one embodiment, obtaining the current state information of the workflow includes at least one of the following: determining a process index matrix to record the process identifiers at each position in the production plan of each machine; determining a process processing time matrix to record the processing time of each process on each machine; determining a process allocation vector to record the processes currently being executed on each machine; determining a process remaining time vector to record the remaining time of the processes currently being executed on each machine; and determining a process dependency matrix to record the dependencies between processes.

[0016] In one embodiment, determining the set of currently executable pending processes based on the current state information and the DAG includes: determining the processes that have not yet been completed in the process subset of the plurality of jobs based on the current state information and the DAG; for each pending process, querying the preceding dependent processes of the process in the DAG, wherein if all the preceding dependent processes of the process have been marked as completed according to the current state information, the process is added to the set of currently executable pending processes.

[0017] In one embodiment, the method further includes: identifying processes of the same type in a subset of processes of different jobs; merging the processes of the same type into a node in the DAG; and when the merged node is executed, processing the same processes in the different jobs simultaneously by the same machine.

[0018] In one embodiment, the method further includes a training process for the reinforcement learning model; during the training process, a pre-trained reinforcement learning model is obtained, and the pre-trained model is fine-tuned according to the number of machines and processes in the target production scenario.

[0019] In one embodiment, it further includes at least one of the following: when a machine failure is detected, setting the processing time of the processes being executed and pending on the failed machine to zero; when an emergency order is detected, adding the new job to the cache queue to wait for loading; when a job is detected to be canceled, marking all incomplete processes in the process subset of the job as completed in the DAG.

[0020] In one embodiment, the reinforcement learning model makes decisions for each machine according to a preset time slot period. In each time slot, the machine that needs to make a decision at the moment decides on the process to be executed from the set of currently executable processes to be processed. The decision space of the reinforcement learning model includes idle actions, which indicate that the machine that needs to make a decision at the moment does not execute a process in the current time slot.

[0021] In one embodiment, the decision-making for each machine further includes: when the machine currently requiring a decision is selected to perform an idle action, the machine does not perform a procedure in the current time slot and re-participates in the decision-making process in the next time slot.

[0022] In one embodiment, the workflow has a preset upper limit on the number of processes; when a new job arrives and the number of processes in the workflow is less than the upper limit, the new job is added to the workflow; and when a new job arrives and the number of processes in the workflow is not less than the upper limit, the new job is added to a cache queue.

[0023] In one embodiment, the method further includes: when a new job is added to the workflow, identifying the processes in the process subset of the new job that are of the same type as the processes in the process subset of the existing job; merging the processes in the process subset of the new job and the processes in the process subset of the existing job that are of the same type into a single node in the DAG, and transferring the original dependencies of the processes in the process subset of the new job and the processes in the process subset of the existing job that are of the same type to the merged node.

[0024] In one embodiment, the reinforcement learning model is trained using a policy gradient method; the DAG is described in at least one of the following ways: a visual graph; an adjacency list; JSON / YAML format; or Cypher query language.

[0025] An electronic device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements any of the methods described above.

[0026] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0027] A computer program product includes a computer program that, when executed by a processor, performs any of the methods described above.

[0028] As can be seen from the above technical solutions, the embodiments of the present invention use DAG to describe the complex dependencies between processes, overcoming the limitations of traditional matrix representation methods in capturing multi-source, branching, or merging structures. This allows for accurate modeling of real manufacturing processes where interactions and shared dependencies exist between operations. Furthermore, through reinforcement learning with continuous state monitoring and reward-based policy updates, this method generates optimized scheduling decisions rather than just feasible solutions, and avoids the local optima problem common in heuristic methods.

[0029] Furthermore, embodiments of the present invention feature dynamic workflow management with capacity constraints and buffer queues, enabling the system to adapt to continuously arriving jobs and handle the dynamic nature of real-world production environments where orders are unpredictable. This contrasts sharply with static scheduling methods that assume all jobs are known in advance. Embodiments of the present invention also support merging similar types of operations for simultaneous processing, leveraging the parallel processing capabilities of modern manufacturing equipment to improve utilization and reduce total production time. Embodiments of the present invention also feature comprehensive state representations, including an operation index matrix, processing time matrix, allocation vector, remaining time vector, and dependency matrix, thereby providing reinforcement learning models with complete context awareness. Compared to simplified state representations, this rich input enables more informed decision-making and faster learning convergence.

[0030] Furthermore, the systematic determination of executable steps based on current state information and DAG dependencies ensures that scheduling decisions adhere to priority constraints. By accurately identifying steps that can begin at each decision moment, infeasible scheduling is prevented, and the model's attention is focused on feasible actions.

[0031] Furthermore, reward calculations based on the number of processes and actual processing time provide clear feedback, encouraging faster job completion while maintaining fairness across jobs of varying complexity. When considering deadlines, the adjusted reward signal helps the model balance efficiency and timeliness. Exception handling mechanisms for machine failures, urgent order insertions, and job cancellations enhance robustness in real-world environments. By handling interruptions without complete rescheduling, the system remains stable and continues to provide effective scheduling decisions even in the event of unforeseen circumstances.

[0032] Furthermore, using pre-trained models with fine-tuning capabilities enables efficient deployment across different production environments. By leveraging general scheduling knowledge while adapting to specific facility characteristics, the amount of data required can also be reduced. Attached Figure Description

[0033] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which will make the above and other features and advantages of the present invention more apparent to those skilled in the art. In the drawings:

[0034] Figure 1 This is a schematic diagram of the matrix representation of the flexible job shop scheduling problem in the prior art.

[0035] Figure 2 Based on Figure 1 A schematic diagram of feasible solutions obtained by the traditional solution method for the problem shown.

[0036] Figure 3 This is a flowchart of a flexible job shop scheduling method according to an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of a DAG used to describe process dependencies in an embodiment of the present invention.

[0038] Figure 5 This is a schematic diagram illustrating the reinforcement learning principle in an embodiment of the present invention.

[0039] Figure 6 This is a schematic diagram of the process index matrix in an embodiment of the present invention.

[0040] Figure 7 This is a schematic diagram of the process time matrix in an embodiment of the present invention.

[0041] Figure 8 This is a schematic diagram of the process allocation vector in an embodiment of the present invention.

[0042] Figure 9 This is a schematic diagram of the remaining time vector of the process in an embodiment of the present invention.

[0043] Figure 10 This is a schematic diagram of the process dependency matrix in an embodiment of the present invention.

[0044] Figure 11 This is a schematic diagram of a process dependency matrix segment of a streaming workflow in an embodiment of the present invention, illustrating how parallel processes in different jobs are integrated when a new job is added.

[0045] Figure 12 This is a schematic diagram of action selection in an embodiment of the present invention.

[0046] Figure 13 This is a schematic diagram of the policy network in an embodiment of the present invention.

[0047] Figure 14 This is a structural diagram of an electronic device according to an embodiment of the present invention.

[0048] The reference numerals in the attached figures are as follows:

[0049] label meaning 100~104 step 10 intelligent agent 20 environment 30 Current status 40 Policy Network 50 probability distribution 800 electronic devices 801 processor 802 memory Specific Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments are provided to further illustrate the invention in detail. The nouns and pronouns referring to "person" in this patent application are not limited to specific genders.

[0051] For the sake of brevity and intuitiveness, the present invention will be described below through several representative embodiments. Numerous details in the embodiments are provided solely to aid in understanding the present invention. However, it is clear that the implementation of the present invention may not be limited to these details. To avoid unnecessarily obscuring the present invention, some embodiments are not described in detail, but only outlines are given. In the following text, "comprising" means "including but not limited to," and "according to..." means "at least according to..., but not limited to only according to...". Due to Chinese language conventions, unless the quantity of a component is specifically indicated below, it means that the component can be one or more, or can be understood as at least one.

[0052] Flexible job shop scheduling (FJSP) is a classic problem in manufacturing systems scheduling. In a flexible job shop, there are multiple jobs and multiple machines. Each job consists of several operations in a specific sequence, and each operation can be processed from a set of available machines. The scheduling objective is usually to minimize the maximum completion time, i.e., the total time from the start of the first job to the completion of the last job. Specifically, flexible job shop scheduling needs to satisfy two types of constraints: priority constraints, i.e., the next operation can only begin after the previous operation in the same job is completed; and resource constraints, i.e., a machine can only process one operation at a time. Unlike traditional job shop scheduling, flexible job shop scheduling allows an operation to be selected from multiple machines, which increases scheduling flexibility but also makes the problem more complex.

[0053] Currently, traditional flexible workshop scheduling methods mainly suffer from the following drawbacks:

[0054] First, the matrix representation method is insufficient in describing dependencies. Traditional methods use matrices to represent the processing time of each operation, with each job corresponding to an independent matrix. This representation can only express linear sequence constraints within a job, meaning that the next operation can only begin after the previous one is completed. However, real-world manufacturing scenarios often involve more complex dependencies, such as sequential constraints between operations in multiple jobs, multiple parallel branches within the same job, or operations from multiple jobs needing to converge at a certain node. The matrix representation method cannot describe these cross-job dependencies and complex structures involving multiple sources, branches, and convergences, leading to significant discrepancies between the scheduling model and reality.

[0055] Second, the quality of the solution is limited. Traditional solution methods, such as heuristic algorithms (e.g., priority allocation rules) or metaheuristic algorithms (e.g., genetic algorithms, tabu search), often only find a feasible solution that satisfies the constraints when faced with complex constraints, and cannot guarantee that it is the optimal solution. These methods are usually based on local search or empirical rules, which easily get stuck in local optima and make it difficult to find a better scheduling scheme globally. As the problem size increases, the gap between the solution quality and the optimal solution may widen further.

[0056] Third, it is difficult to adapt to dynamically changing work environments. Traditional methods typically assume that all jobs are known and fixed at the start of scheduling, and the scheduling plan is not adjusted once generated. However, in actual production, new jobs may arrive at any time, urgent orders may be inserted, existing jobs may be canceled, and machines may malfunction. The static scheduling model of traditional methods cannot cope with such dynamic changes. When the environment changes, a complete recalculation of the scheduling is required, which is computationally expensive and has a slow response time.

[0057] Fourth, the parallel processing capabilities of modern equipment were not fully utilized. Traditional methods strictly adhere to the assumption that "one machine can only process one operation at a time," with all operations strictly sequential on the timeline. However, modern manufacturing equipment, such as large heat treatment furnaces, painting booths, and multi-station machining centers, often have the ability to process multiple identical operations simultaneously. Traditional methods fail to identify and utilize these parallel opportunities, leading to wasted equipment capacity and limiting the improvement of production efficiency.

[0058] Fifth, it lacks self-optimization capabilities. Traditional methods stop once a scheduling scheme is generated, and cannot learn from historical scheduling experience to improve. Faced with similar scheduling scenarios, traditional methods will repeat the same calculation process, instead of using past experience to obtain better scheduling strategies.

[0059] Figure 1 This is a schematic diagram of the matrix representation of the flexible job shop scheduling problem in existing technologies. For example... Figure 1 As shown, this example includes 3 machines (M1, M2, M3) and 3 jobs (Job 1, Job 2, Job 3), each job consisting of 3 consecutive operations. Wherein, O jk This represents the k-th step of the j-th job, for example, O 11 O represents the first step of Operation 1. 12 O represents the second step of Operation 1. 13 This indicates the third step in Operation 1. Figure 1 The three matrices shown correspond to three jobs, and the numbers in the matrices represent the processing time for each process on different machines. For example, in the matrix for job 1, the first row represents process O. 11 The processing time on machines M1, M2, and M3; the second line indicates operation O. 12 Processing time on machines M1, M2, and M3; the third line indicates operation O. 13 Processing time on machines M1, M2, and M3.

[0060] However, in Figure 1 In the matrix representation shown, each job uses an independent matrix, and the matrices lack correlation, failing to express the interrelationships between different job processes. For example, when multiple job processes need to share the same machine, the scheduling constraints resulting from this resource sharing cannot be reflected in the matrix. Furthermore, the matrix representation can only express the linear order within a job (such as O...). 11 →O 12 →O 13 However, it cannot describe more complex dependency structures, such as when a process needs to wait for multiple preceding processes to complete (merging structure), or when multiple processes can be executed in parallel (branching structure). This limitation in representation leads to a discrepancy between the scheduling model and reality.

[0061] Figure 2 Based on Figure 1 The diagram illustrates a feasible solution obtained using traditional methods for solving the problem, with a Gantt chart showing the time allocation of each process on the machine. Figure 2 In the diagram, the horizontal axis represents time from 0 to 6, the vertical axis represents the machine, each rectangle represents a process, the length of the rectangle represents the processing time of that process, and the position of the rectangle represents the start and end time of that process on the corresponding machine.

[0062] like Figure 2 As shown, on machine M1, process O 21 Processing from time 0 to time 2, step O 22 Processing from time 2 to time 3, step O 12 Processing from time 3 to time 5. On machine M2, process O... 31 Processing from time 0 to time 1, step O 11 Processing from time 1 to time 2, step O 23 Processing from time 3 to time 4, step O 33 Processing from time 5 to time 6. On machine M3, process O... 32 Processing from time 1 to time 5, step O 13 Processing begins at time 5 and continues until time 6. The total completion time for this scheduling scheme is 6 time units.

[0063] However, Figure 2 The scheduling scheme shown is only one of many feasible solutions, and its corresponding total completion time may not be optimal. By adjusting the allocation of processes across different machines or changing the execution order of processes, a better solution with a shorter total completion time may exist, but traditional methods cannot guarantee finding the globally optimal solution. Furthermore, Figure 2 The scheduling scheme shown assumes that all jobs are known and fixed at the start of scheduling, and cannot handle changes such as the dynamic arrival of new jobs or job cancellations. If a new job arrives at this time, the entire scheduling scheme may need to be recalculated. Moreover, as... Figure 2 As shown, traditional methods strictly adhere to the assumption that "a machine can only process one operation at a time," with all operations strictly sequential on the timeline. When multiple jobs require the same processing, they could ideally be processed simultaneously on the same equipment, but... Figure 2 These were scheduled at staggered times, resulting in wasted equipment capacity. Furthermore, Figure 2 The scheduling scheme shown is fixed once generated and cannot be improved from scheduling experience. When encountering a similar situation again... Figure 1 When dealing with similar sets of jobs, traditional methods will repeat the same computation process and will not be able to leverage past experience to obtain a better scheduling strategy.

[0064] To address the shortcomings of the existing technology, this invention proposes a flexible job shop scheduling method based on reinforcement learning and constrained by a Directed Acyclic Graph (DAG). The core idea of ​​this invention is as follows:

[0065] First, regarding dependency description, this invention abandons the traditional matrix representation and introduces a Directed Acyclic Graph (DAG) as a unified description framework. In this framework, the dependencies between all job operations are described by a DAG, and each job contains a subset of its corresponding operations within that DAG. Nodes in the DAG represent operations, and directed edges represent dependencies between operations, naturally expressing complex structures such as multi-source (multiple initial operations), branching (parallel operations), and merging (subsequent operations can only begin after multiple operations are completed). This fundamentally solves the problem of insufficient dependency description capabilities in traditional methods.

[0066] Secondly, regarding the construction of state information, the embodiments of this invention can obtain a comprehensive set of information reflecting the workflow state, including: a process index matrix (recording the process identifiers at each position in the production plan of each machine), a process processing time matrix (recording the processing time of each process on each machine), a process allocation vector (recording the processes currently being executed on each machine), a process remaining time vector (recording the remaining time of the processes currently being executed on each machine), and a process dependency matrix (recording the dependencies between processes). These five types of information together constitute the current state information required for the reinforcement learning model's decision-making.

[0067] Furthermore, regarding the decision-making and optimization mechanism, this embodiment of the invention models the scheduling problem as a reinforcement learning problem. The core elements include: a decision-making agent (the reinforcement learning model acts as an agent, making decisions for each machine according to a preset time slot period), a state space (the current state information composed of the above five types of information), an action space (including selecting specific procedures to execute and selecting idle actions, where idle actions indicate that the machine does not execute procedures in the current time slot), a reward function (when a job is completed, a reward value is calculated based on the number of procedures and the actual processing time; for jobs with deadlines, the reward value is also adjusted based on the deadline), and a strategy update (continuously updating the reinforcement learning model parameters based on the reward value, thereby continuously optimizing the scheduling strategy). Through the reinforcement learning framework, this embodiment of the invention can continuously learn and optimize scheduling decisions in a dynamically changing work environment, overcoming the shortcomings of traditional methods, such as limited solution quality and lack of self-optimization capabilities.

[0068] Furthermore, regarding parallel processing, this embodiment of the invention identifies processes of the same type in different jobs and merges them into a single node in the Directed Acyclic Graph (DAG). When this merged node is executed, these identical processes are processed simultaneously by the same machine. This mechanism fully utilizes the parallel processing capabilities of modern equipment to handle multiple identical processes simultaneously, effectively improving equipment utilization and production efficiency. Simultaneously, regarding dynamic adaptability, the workflow of this embodiment of the invention has a preset upper limit on the number of processes and continuously monitors the arrival of new jobs. When a new job arrives and the workflow is not full, it is directly added to the workflow; when the workflow is full, the new job is added to a cache queue to await loading. This mechanism enables this embodiment of the invention to readily handle dynamic changes such as the dynamic arrival of new jobs and the insertion of urgent orders.

[0069] Finally, regarding exception handling, this embodiment of the invention incorporates mechanisms for handling common anomalies: when a machine malfunction is detected, the processing time of currently executing and pending operations on the malfunctioning machine is set to zero; when an urgent order is detected, the new job is added to the cache queue to await loading; when a job is detected to be canceled, all incomplete operations in the operation subset of that job are marked as completed in the DAG. These exception handling mechanisms enhance the robustness of this invention in real-world production environments.

[0070] Through the overall design of the above aspects, the embodiments of the present invention effectively solve the problems of insufficient dependency description capability, limited solution quality, difficulty in adapting to dynamic changes, failure to utilize parallel processing capability, and lack of self-optimization capability in the prior art. It provides a flexible job shop scheduling method that can support complex dependencies, dynamic working environment, parallel processing, and has continuous optimization capability.

[0071] Figure 3 This is a flowchart of a flexible job shop scheduling method according to an embodiment of the present invention. Figure 3 As shown, the method includes the following steps:

[0072] Step 101: Obtain the current status information of the workflow, which includes multiple jobs and multiple machines. The dependencies between the procedures of the multiple jobs are uniformly described by DAG. Each job contains a subset of procedures in the DAG. Each procedure corresponds to at least one optional machine. Each optional machine is selected from multiple machines.

[0073] Here, the first step is to obtain the current status information of the workflow. A workflow contains all jobs in the current production system and all machines available to process those jobs. Essentially, it's a real-time view of the entire shop floor, showing which jobs are waiting, which processes are running, and which resources are available. Workflows have limited capacity, meaning they can only process a certain number of processes at any given time. Current status information can include the following data: which machines are idle or busy, which processes are currently being processed on each machine, how much time remains for each ongoing process, which processes have been completed, and which jobs are waiting to be processed.

[0074] A job represents a production task that needs to be completed. For example, in an automobile factory, one job might be "manufacturing the car body," while another might be "manufacturing the car door." Each job consists of multiple processes, which need to be executed in a specific order based on their dependencies. Machines are the physical resources that execute these processes. Different types of machines handle different types of processes: a cutting machine handles cutting, a welding robot handles welding, a heat treatment furnace handles heat treatment, and so on. Dependencies between processes define the priorities that must be followed during production. For example, a car door cannot be welded before it is cut to the correct shape; a car body cannot be heat-treated before it is welded. These dependencies can be simple linear sequences or form more complex structures, such as multiple processes that can be performed in parallel, or multiple processes that must all be completed before a subsequent process begins. All these dependencies are uniformly described by a Directed Acyclic Graph (DAG). Each node represents a process, and each directed edge from node A to node B indicates that process A must be completed before process B begins. The acyclic property ensures that there are no circular dependencies; there will not be a situation where process A depends on B, B depends on C, and C depends on A. A DAG provides a unified framework that captures the complete dependency structure of all jobs in a factory.

[0075] Each job comprises a subset of operations within the global Directed Acyclic Graph (DAG). In other words, a job is not a single linear sequence, but rather a specific set of nodes in the overall graph, along with the edges connecting these nodes. For example, the global DAG might contain twenty different types of operations, but a particular car body manufacturing job might only require four: cutting, welding, heat treatment, and polishing. These four operations and their specific dependencies constitute the operation subset of the job. Each operation corresponds to at least one optional machine, selected from the pool of available machines. An operation can be performed on any of the optional machines, and different machines may require different times to complete the same operation. For example, a cutting operation might be performed on either cutting machine M1 or cutting machine M2, with cutting machine M1 taking 30 minutes and cutting machine M2 taking 35 minutes. Each optional machine comes from the set of machines in the workflow.

[0076] Example: Consider a small factory with three machines: a cutting machine M1, a welding machine M2, and a heat treatment furnace M3. Currently, two jobs are active in the workflow: Job A (car body) requires three sequential steps: cutting A1, welding A2, and heat treatment A3; Job B (car door) requires two sequential steps: cutting B1 and welding B2. The current production status is as follows: Cutting machine M1 is available; welding machine M2 is performing welding step A2, with 10 minutes remaining; heat treatment furnace M3 is available. Regarding process progress: A1 is complete, B1 is complete, A2 is in progress (10 minutes remaining), while steps A3 and B2 have not yet started.

[0077] The acquired current status information captures all of this: the status of each machine (idle or busy and remaining time), the completion status of each process, and the relationships between jobs, processes, and machines. This complete view of the current situation provides the basis for intelligent scheduling decisions in subsequent steps.

[0078] Step 102: Based on the current state information and DAG, determine the set of currently executable processes to be processed.

[0079] Based on the current state information and the Directed Acyclic Graph (DAG), identify which operations can begin at the current moment. Not all operations that have not yet started can begin immediately; some must wait for their predecessor operations to complete. An operation is considered currently executable if and only if all its predecessor operations have been completed. Predependent operations are operations that must be completed before this operation can begin, defined by the incoming edges in the DAG. For example, if there is an edge in the DAG pointing from operation X to operation Y, then X is a predecessor operation of Y, and Y must wait until X is completed before it can begin.

[0080] To determine the set of currently executable steps, each step that is not yet completed can be examined. For each such step, the Directed Acyclic Graph (DAG) is queried to identify all its predecessor steps. The current state information is then checked to verify that all these predecessor steps are marked as completed. Only when every predecessor step has been confirmed as completed does the step become part of the current set of executable steps.

[0081] Recalling the previous example, processes A1 and B1 are complete, A2 is in progress (10 minutes remaining), and processes A3 and B2 have not yet started. For process A3: querying the DAG reveals its predecessor process is A2. Checking the current status information shows A2 is still in progress (not yet complete). Therefore, A3 is currently not executable. For process B2: querying the DAG reveals its predecessor process is B1. Checking the current status information shows B1 has been marked as complete. Therefore, B2 is currently executable.

[0082] Therefore, at this point, the set of executable processes only contains process B2. This information will be passed to the reinforcement learning model in subsequent steps.

[0083] Step 103: Input the current state information and the set of processes to be processed into the reinforcement learning model, so that the reinforcement learning model can decide the process to be executed in this operation from the set of processes to be processed, and allocate a machine to the process from at least one optional machine corresponding to the process to be executed in this operation.

[0084] A reinforcement learning model is invoked to make scheduling decisions. This model determines which process should be executed next and which machine should handle it. A reinforcement learning model is an artificial intelligence model that has been trained (or is being trained) to make optimal scheduling decisions. It learns from experience: it receives positive reinforcement when its decisions lead to good results (such as quick job completion) and negative reinforcement when its decisions lead to bad results. Over time, it learns to favor actions that result in better overall scheduling performance. The model operates according to a pre-set time slot cycle. At fixed time intervals (e.g., every minute) or whenever a machine becomes idle, the model is invoked to make decisions for the machine that currently needs guidance. This periodic decision-making ensures that the system can respond quickly to changes in the production floor.

[0085] The model's action space includes all possible choices it can make. In this embodiment, the action space includes two types:

[0086] (1) Execution of operations: The model can select any operation from the current set of executable operations and assign it to the appropriate machine. For each selected operation, the model must also select from the set of available machines for that operation which machine will be used for the operation.

[0087] (2) Idle action: The model can decide whether a machine should remain idle in the current time slot. This may be the optimal choice when there is no suitable executable operation for the machine, or when waiting for a high-priority operation that is about to become executable will bring better long-term results.

[0088] The model's decisions balance short-term considerations with long-term optimization. For example, it might choose to start a less urgent process now if doing so would result in better machine utilization; or it might deliberately leave a machine idle in anticipation of a high-priority process that will become available soon.

[0089] Continuing from the previous example, the current set of executable operations only contains B2. The machines that need to make a decision are M1 (idle) and M3 (idle). M2 is still processing A2. The reinforcement learning model considers the current situation: (1) For M1: B2 is a welding operation, but M1 is a cutting machine and cannot perform welding. The only available machine for B2 is M2 (busy), and in this example, only M2 can weld. Therefore, M1 has no available executable operations. The model selects an idle action for M1. (2) For M3: B2 is a welding operation, but M3 is a heat treatment furnace and cannot weld. There is no suitable executable operation for M3. The model selects an idle action for M3.

[0090] Therefore, during this time slot, M1 and M3 remain idle, while M2 continues processing A2. The system will wait for the next time slot to make a new decision.

[0091] Step 104: Execute the process to be performed on the assigned machine.

[0092] Execution means that each machine assigned a process begins processing. During execution, the machine remains busy during the processing time of its process. As time progresses, processes are completed, and the workflow status is updated accordingly. Throughout this process, the method continuously monitors the completion of jobs, tracks the progress of each process, and continuously checks whether any job has completed all processes.

[0093] It should be noted that the tasks described in steps 101-104 can be actual production tasks in a real production environment, used to execute the pre-trained reinforcement learning model for scheduling decisions. The reinforcement learning model in this embodiment needs to be obtained in advance through a training process, which is the same as the decision mechanism in steps 101-104, but the training process can use training sample tasks instead of actual production tasks. During the training of the reinforcement learning model, when all processes included in any training sample task are completed, a reward value is calculated based on the completion information of that training sample task. The completion information includes the number of processes included in the training sample task and the actual processing time from the first process to the last process. In a preferred embodiment, the reward is calculated as the number of processes divided by the actual processing time. For training sample tasks with deadlines, the reward value can be further adjusted: a positive reward is given for early completion, zero reward for on-time completion, and a negative reward for late completion. The parameters of the reinforcement learning model are updated based on the calculated rewards, making the model more inclined to produce high-reward behaviors in subsequent decisions. Through repeated iterations with a large number of training sample tasks, the model gradually learns an optimized scheduling strategy. Once training is complete, the model can be deployed to actual production. At this point, reward calculations and model updates will no longer be performed after step 104; only the scheduling of actual production jobs will be executed.

[0094] This example demonstrates how to train a reinforcement learning model using training sample jobs. At t=100, step C1 of training sample job C is completed, and step C2 of training sample job C becomes executable. At this time, the system recognizes that C2 and step A3 of training sample job A are both heat treatment steps of the same type, triggering the same-type step merging mechanism. In the DAG, steps C2 and A3 are merged into a heat treatment merge node, and the original dependencies between steps C2 and A3 are transferred to this merge node. Since the prerequisites A2 and C1 of the merge node have been satisfied, the merge node becomes executable. The model decides to execute this merge node for machine M3, which simultaneously processes the heat treatment steps of training sample jobs A and C. After several time slots, M2 completes A2, the currently executable step set is updated, and A3 becomes executable. In subsequent decision slots, the model assigns A3 to M3 and assigns B2 to M2 when machine M2 becomes available. Finally, B2 completes, at which point all steps of training sample job B have been executed. The system detects the completion of training sample job B and calculates a reward value based on its completion information: the number of processes is 2, and the actual processing time from the first process B1 to the last process B2 is 95 minutes. The reward value is calculated as 2 / 95≈0.021, and the parameters of the reinforcement learning model are updated based on this reward value. This positive reward reinforces the decision sequence that led to the completion of the training sample job. Subsequently, when the last process A3 of training sample job A is completed, training sample job A is also completed. The system calculates a reward value for it and uses it for further model updates. Through repeated iterations with a large number of training sample jobs, the model parameters are gradually optimized, eventually converging to obtain the optimal scheduling strategy. At this point, the model training is complete and it can be deployed for scheduling actual production operations.

[0095] During the training of a reinforcement learning model, for each training sample job, a reward value needs to be calculated based on its completion information as a feedback signal for model updates. The reward calculation method is explained in detail below with reference to specific embodiments. In one embodiment, calculating the reward value based on the completion information of a training sample job includes: obtaining the number of processes contained in the process subset of the training sample job; obtaining the actual processing time of the training sample job from the start of the first process to the completion of the last process; and calculating the reward value based on the number of processes and the actual processing time.

[0096] Specifically, when all operations within a training sample job are completed, the system first obtains the number of operations contained in the subset of operations of that training sample job. Each training sample job is defined by its subset of operations in the DAG. When all these operations have been completed, the system records the number of operations required to complete the training sample job. For example, a training sample job requiring cutting, welding, heat treatment, and polishing has four operations; a training sample job requiring only cutting and welding has two operations. The number of operations reflects the complexity and workload involved in the training sample job. The system then obtains the actual processing time of the training sample job, which is the total time elapsed from the start of the first operation to the completion of the last operation. It should be noted that the actual processing time is not simply the sum of the processing times of each operation, as there may be waiting time between operations due to machine availability or scheduling decisions. The actual processing time accurately reflects the total time taken for the training sample job from entering active processing to complete completion. For example, in the training phase example above, the first process B1 of training sample job B starts at t=10 and the last process B2 is completed at t=85, with an actual processing time of 75 minutes. Even though the sum of the processing times of all processes is only 30+45=75 minutes, there is no waiting time. However, the first process A1 of training sample job A starts at t=30 and the last process A3 is completed at t=160, with an actual processing time of 130 minutes, which includes the waiting time between processes.

[0097] Based on the acquired number of processes and the actual processing time, the system calculates a reward value. In a preferred embodiment, the reward value is calculated as the number of processes divided by the actual processing time. This calculation formula has an intuitive physical meaning: it rewards training sample jobs that are completed faster relative to their complexity. As described in claim 5, for training sample jobs with the same number of processes, the shorter the actual processing time, the higher the reward; while for the same actual processing time, training sample jobs with more processes receive a higher reward, reflecting the higher value of completing complex tasks.

[0098] Consider a concrete example: two training tasks are completed simultaneously. Training task X requires three steps and takes 90 minutes from start to finish. Training task Y also requires three steps but takes only 60 minutes. Using the reward calculation method of dividing the number of steps by the actual processing time, training task X receives a reward of 3 / 90 = 0.0333, while training task Y receives a reward of 3 / 60 = 0.05. The reinforcement learning model learns from this result that the decision sequence that leads to faster completion of training task Y is more effective.

[0099] Consider a more complex example: Training task Z requires five steps and takes 100 minutes to complete, with a reward of 5 / 100 = 0.05, the same as the reward for training task Y. This shows that completing a more complex training task in a longer time may have the same value as completing a simpler one in a shorter time, depending on the ratio between the two.

[0100] The above reward calculation method has the following advantages: First, it provides continuous, fine-grained reward signals, rather than simple binary success / failure indicators, providing richer feedback information for model learning; Second, by normalizing by dividing by the actual processing time, it encourages the model to focus not only on minimizing process execution time but also on reducing waiting time between processes, thereby promoting the improvement of overall workflow efficiency; Third, using the number of processes as the numerator ensures that training sample jobs with a large number of processes (usually representing higher-value production outputs) receive appropriate rewards; Fourth, the calculation method is simple and direct, with low computational overhead, and does not significantly burden the training process; Fifth, the reward scale can adapt to different job types and scales, providing consistent feedback signals in diverse training scenarios.

[0101] In training a reinforcement learning model, in addition to the basic reward calculation based on the number of processes and actual processing time, deadline information can be introduced to adjust the reward value, enabling the model to learn scheduling strategies that meet time constraints. In one embodiment, the training process also includes: obtaining the deadline for training sample jobs where all processes have been completed; adjusting the reward value based on the deadline, where early completion earns a positive reward, on-time completion earns zero reward, and late completion earns a negative reward. By integrating deadline information into the training sample jobs, the reward calculation is further refined, enabling the reinforcement learning model to learn to optimize scheduling decisions while meeting time constraints. Specifically, the method first obtains the deadline of the recently completed training sample job. The deadline is the expected completion time of the training sample job, which may come from various simulation scenarios, such as simulated customer order delivery dates, simulated downstream production process dependencies, or simulated internal production target time limits. The deadline is usually expressed as an absolute time or a time offset from when the training sample job enters the system. For example, a training scenario might require the training sample job to be completed within 120 minutes of starting.

[0102] After obtaining the actual processing time of the training sample job, the method compares it with the deadline to determine whether the training sample job is completed ahead of time, on time, or late. The comparison produces three possible results: if the training sample job is completed before the deadline, it is completed ahead of time; if it is completed exactly on the deadline, it is completed on time; if it is completed after the deadline, it is completed late. Based on this judgment, the method adjusts the base reward value calculated from the number of operations and the actual processing time, applying a correction factor that reflects the timeliness performance: (1) For early completion, the reward is adjusted to a positive value. This means that regardless of the base reward value, completion before the deadline always receives positive reinforcement, thereby encouraging the decision sequence that leads to this result. In a preferred embodiment, the base reward is retained and may be further enhanced to reinforce the behavior of early delivery. (2) For on-time completion, the reward is adjusted to zero or remains unchanged. This indicates that meeting the deadline exactly is acceptable, but does not provide additional positive reinforcement. The model learns that on-time performance is neutral, neither particularly encouraged nor penalized. (3) For late completion, the reward is adjusted to a negative value. This penalizes decision sequences that lead to missed deadlines, even if the training sample assignments are eventually completed. Negative rewards prevent the model from repeating decision patterns that result in delays. In a preferred embodiment, the base reward is inverted or negative to create a negative value that varies with the degree of delay, with greater penalties for more severe delays.

[0103] In training reinforcement learning models, reward calculation follows a fundamental principle: for training sample tasks of comparable complexity, the faster the completion speed, the higher the reward. This principle drives the model to learn a scheduling strategy that minimizes processing time relative to task complexity. Specifically, the reward value is calculated based on two key pieces of information: the number of steps in the subset of steps that have been completed in the training sample task, and the actual processing time from the start of the first step to the completion of the last step in that training sample task. In a preferred embodiment, this principle is implemented by calculating the reward as the number of steps divided by the actual processing time. Using this formula, the reward is directly proportional to the number of steps and inversely proportional to the processing time. For two training sample tasks with the same number of steps, the task with the shorter processing time produces a larger quotient and therefore receives a higher reward.

[0104] Figure 3 The steps shown form a complete closed-loop scheduling system. Step 101 captures the current state of the production workshop. Step 102 identifies which processes can begin. Step 103 uses a reinforcement learning model to determine the specific task to be performed next. Step 104 executes the decision. This loop repeats continuously in each time slot, enabling the scheduling system to adapt to changing conditions and continuously improve its performance over time.

[0105] In one embodiment, the current state information is determined by at least one of the following methods: (1) determining a process index matrix to record the process identifier at each position in the production plan of each machine; (2) determining a process processing time matrix to record the processing time of each process on each machine; (3) determining a process allocation vector to record the processes currently being executed on each machine; (4) determining a process remaining time vector to record the remaining time of the processes currently being executed on each machine; and (5) determining a process dependency matrix to record the dependencies between processes. These five data structures together provide a comprehensive representation of the workflow state, enabling the reinforcement learning model to make informed scheduling decisions.

[0106] The process index matrix records the process identifiers at each position in the production plan of each machine. This matrix has dimensions N×M, where M is the number of machines, N represents the workflow capacity, and the maximum number of processes that each machine can queue or plan. The element in the nth row and mth column represents the process arranged at that position in the production plan of machine m. For example, in a factory with three machines, each with a workflow capacity of five processes, the process index matrix might show that process A1 is arranged at position 1 of machine M1, process B2 is arranged at position 2 of machine M2, and empty positions are marked accordingly. This matrix clearly shows the processes that each machine is expected to process and their sequence, laying the foundation for understanding machine workload and process sequencing.

[0107] The process time matrix records the processing time of each process on each machine. This matrix also has dimensions N×M and corresponds positionally to the process index matrix. For each process identified in the process index matrix, the corresponding element in the processing time matrix indicates how long that process will take on its assigned machine. For example, if process A1 takes 30 minutes on M1, the corresponding entry will be 30; if a process cannot be processed on a specific machine, or if no process is scheduled, the entry is zero. This matrix provides the necessary information for predicting process durations and calculating completion times.

[0108] The job assignment vector records the jobs currently being executed by each machine. This is a 1×M vector, where each element represents the job currently in progress on the corresponding machine. For example, at any given moment, the vector might show that machine M1 is executing job A2, machine M2 is executing job B1, and machine M3 is idle. This vector provides a real-time snapshot of which jobs are being actively processed, which is crucial for determining when machines are available and which resources are currently occupied. Unlike traditional scheduling representations, the job assignment vector allows multiple identical job indices to appear simultaneously, reflecting that jobs may have multiple sources and branches; for example, different parts of the same car body may be processed in parallel on different machines.

[0109] The process remaining time vector records the remaining processing time for each process currently being executed on each machine. This is also a 1×M vector, corresponding positionally to the process assignment vector. For each machine, this value represents how much time remains before the current process is completed. Continuing the previous example, if process A2 on machine M1 has 15 minutes remaining and process B1 on machine M2 has 10 minutes remaining, the vector will display these values ​​accordingly. This vector provides crucial timing information for predicting when machines will become idle and when new processes can begin.

[0110] The process dependency matrix records the dependencies between processes in a workflow. It is an N×N matrix where the element in the u-th row and v-th column indicates whether the start of process u depends on the completion of process v; a value of 1 indicates dependency, and 0 indicates no dependency. For example, consider a DAG where process A3 depends on A1 and A2, and process B2 depends on B1. In the dependency matrix, this would be represented by setting 1 in the appropriate position and 0 in the others. This matrix encodes the complete dependency structure of the workflow in a format easily processed by reinforcement learning models, enabling them to understand which processes must wait for other processes to complete before they can begin.

[0111] These five data structures collectively provide a complete picture of the workflow state. The process index matrix indicates what processes are planned for each machine, the process processing time matrix indicates how long each planned process will take, the process assignment vector indicates what is currently running, the process remaining time vector indicates when the running process will finish, and the process dependency matrix indicates which processes must wait for other processes. First, matrix-based representations can be efficiently processed by neural networks because these data structures can be directly input into the model without extensive preprocessing. Separating different types of information into different matrices and vectors allows the model to learn specialized feature extractors for each aspect of the state, improving learning efficiency and decision quality. Using fixed-size matrices with zero padding allows the model to handle workflows of varying sizes within a consistent input format, simplifying the model architecture. The process dependency matrix explicitly encodes the DAG structure in a machine-readable format, enabling the model to learn complex dependency patterns without separately parsing the graph structure. Finally, including remaining time information gives the model time-awareness, allowing it to predict future events and make decisions that consider both current and future consequences.

[0112] In one embodiment, determining the set of currently executable pending processes based on current state information and a Directed Acyclic Graph (DAG) includes: identifying incomplete processes within a subset of processes from multiple jobs based on the current state information and the DAG; for each incomplete process, querying the DAG for its predecessor dependent processes, wherein if all predecessor dependent processes for the process are marked as completed based on the current state information, the process is added to the set of currently executable pending processes. In this embodiment, the process of determining the set of currently executable pending processes based on current state information and the DAG involves a systematic evaluation of the readiness state of each process. Determining the set of currently executable pending processes further includes: identifying incomplete processes based on the current state information. The current state information records the completion status of each process in each job, including not started, in progress, or completed. Based on the completion status, processes not yet marked as completed are identified as pending processes. For each pending process, the DAG is queried to determine its predecessor dependent processes. The predecessor dependent processes refer to processes that must be completed before the pending process begins, defined by directed edges in the DAG. For example, if a directed edge exists in the DAG pointing from process X to process Y, then process X is a prerequisite process for process Y. In scenarios with multiple prerequisites, such as an assembly process that requires the completion of multiple component processes before it can begin, the DAG will contain multiple directed edges pointing to that assembly process. Using this method, processes that satisfy all prerequisite conditions at the current moment can be accurately identified, thereby generating the set of currently executable processes to be processed.

[0113] Consider a simple manufacturing scenario with three processes: A1 (cutting), A2 (welding), and A3 (heat treatment), where the Directed Acyclic Graph (DAG) specifies that A2 depends on A1, and A3 depends on A2. At a certain point, the current status information shows that A1 is completed, A2 is in progress, and A3 has not yet started. First, identify the processes that are not yet completed. These are A2 (still in progress) and A3 (not yet started). For process A2, query the DAG and find that its prerequisite is A1. Checking the current status information confirms that A1 is completed. Therefore, A2 is added to the set of currently executable processes, even though it is already in progress; it was executable at the start, and this determination helps track what can start now if the machine is available. For process A3, query the DAG and find that its prerequisite is A2. Checking the current status information shows that A2 is still in progress and not completed. Therefore, A3 will not be added to the set of currently executable processes. It must wait until A2 is completed before it can start.

[0114] The method for determining executable actions offers several advantages. First, it ensures that all scheduling decisions respect the dependency constraints encoded in the DAG, preventing infeasible scheduling where actions begin before their prerequisites are met. Second, by systematically evaluating each pending action based on its pre-determined dependencies, this method guarantees the integrity and accuracy of the set of executable actions at each decision point. Furthermore, the separation of completion state tracking (via current state information) from dependency definition (via the DAG) allows the system to handle dynamic changes; when an action completes, the current state information is updated, and the set of executable actions is automatically recalculated in the next decision cycle. Fourth, this method scales efficiently even with complex DAG structures because the dependencies of each action are pre-encoded and can be quickly retrieved. Finally, by providing the reinforcement learning model with a precisely defined set of currently feasible actions, this method focuses the model's attention on feasible choices, improving learning efficiency and decision quality.

[0115] In one embodiment, the method further includes: identifying processes of the same type in a subset of processes of different jobs; merging processes of the same type into a node in the DAG; and when the merged node is executed, processing the same processes in different jobs simultaneously by the same machine.

[0116] In this embodiment, the method further improves productivity by identifying and utilizing opportunities for parallel processing. Modern manufacturing equipment, such as large heat treatment furnaces, painting booths, or multi-station machining centers, typically have the capability to process multiple identical processes simultaneously. This embodiment provides a mechanism for identifying and utilizing such opportunities within a scheduling framework. First, processes of the same type are identified across different jobs. The process type is determined by the nature of the required processing, such as heat treatment, painting, or curing. Even if these processes belong to different jobs and may be located at different points in their respective job sequences, they are candidates for merging if they require the same type of processing. For example, a body manufacturing job may require heat treatment after welding, while an engine part job may require heat treatment after machining. Both processes are heat treatment types and could potentially be processed together. Once such identical processes are identified across different jobs, the method modifies the Directed Acyclic Graph (DAG) to reflect this merging opportunity. Specifically, identical processes are merged into a single node in the DAG. This merge node represents the collective processing of all these individual processes as a unified task. The dependencies of each original process are preserved and transferred to this merge node. This means that the merge node inherits all the preceding dependencies from each original process, and all the subsequent dependencies from each original process now depend on the completion of the merge node.

[0117] For example, suppose process A3 depends on the completion of A2, and process C2 depends on the completion of C1. After merging, a single node is created: the heat treatment batch. This node depends on the completion of both A2 and C1, reflecting that these two preconditions must be met before batch processing can begin. After batch processing is complete, A4 and C3 can begin; they depend on the completion of their respective heat treatment processes, now represented by the merged node. When the reinforcement learning model selects to execute the merged node, the method processes all merged processes simultaneously on a single machine. This means that instead of scheduling separate time slots for each individual heat treatment process, the machine processes them together as a batch. The processing time of the merged node can be determined based on the batch size and machine capacity; typically, it might be the time required to process the largest process, or a base time that scales moderately with the batch size, reflecting the efficiency gains of batch processing.

[0118] Consider a factory with a heat treatment furnace capable of processing multiple components simultaneously. There are three jobs in the workflow: job A requires heat treatment as its third step; job C requires heat treatment as its second step; and job E also requires heat treatment as its second step. This method identifies that jobs A3, C2, and E2 are all of the same type: heat treatment. It merges them into a node in the Directed Acyclic Graph (DAG) called a heat treatment batch. This merge node inherits dependencies: it requires that A2, C1, and E1 have all been completed before the batch can begin. When the reinforcement learning model decides to execute the heat treatment batch node, the furnace processes all three components simultaneously in a batch. If heat treating each component individually takes 60 minutes, the batch process might only take 80 minutes in total, compared to 180 minutes if processed individually, significantly improving efficiency. After the batch process is complete, jobs A4, C3, and E3 become ready to begin.

[0119] The method of merging similar processes offers significant advantages. First, it significantly improves equipment utilization by enabling batch processing, reducing idle time on expensive machinery. Furthermore, it reduces total completion time by processing multiple processes in parallel rather than sequentially, potentially shortening overall production time. Additionally, it reduces energy consumption and operating costs associated with repetitive equipment setup and preheating cycles. Fourth, by merging processes in a DAG, it simplifies the scheduling problem; fewer nodes mean a smaller decision space for the reinforcement learning model.

[0120] In one embodiment, the training process of a reinforcement learning model includes a pre-training phase and a fine-tuning phase. The pre-training phase refers to the initial training of the model on a wide variety of general scheduling scenarios. This phase uses a large-scale general scheduling dataset, covering production scenarios with various machine configurations, job types, job structures, and scales. Through pre-training, the model learns universal scheduling strategies, such as prioritizing critical processes that can unlock multiple downstream tasks, evenly distributing workloads across different machines, and selecting idle actions to wait for higher-priority tasks when appropriate. The pre-training phase can employ the training mechanism described above, calculating a reward value and updating the model parameters based on completion information after all processes of each general training sample job have been completed, or it can use other reinforcement learning algorithms or supervised learning methods for training.

[0121] The fine-tuning phase refers to the targeted adjustment of the model based on pre-training data from the target production scenario, adapting it to the specific application environment. Key parameters for fine-tuning include the number of machines and processes in the target scenario, which determine the basic scale and complexity of the scheduling problem. The fine-tuning phase can employ the training mechanism described above, training sample jobs for each target scenario, calculating reward values ​​based on completion information and updating model parameters after all processes are completed; alternatively, other fine-tuning methods can be used, such as adjusting only the parameters of the later layers of the network. Unlike the pre-training phase, the data used in the fine-tuning phase comes from the target production scenario, and the starting point for updating model parameters is the pre-trained model rather than random initialization.

[0122] In summary, pre-training and fine-tuning are two specific stages in the training process. Pre-training uses general data to learn a general policy, while fine-tuning uses scenario-specific data to adapt to a specific environment. Pre-training provides good initial parameters for fine-tuning, enabling the fine-tuning stage to obtain a high-performance scenario-adapted model with less data and faster convergence.

[0123] In one embodiment, the method further includes at least one of the following: when a machine failure is detected, setting the processing time of all ongoing and pending operations on the failed machine to zero; when an urgent order is detected, adding the new job to a cache queue to await loading; and when a job cancellation is detected, marking all incomplete operations in the operation subset of the job as completed in the DAG. In this embodiment, the method further incorporates a robust exception handling mechanism to address common disruptions occurring in real-world manufacturing environments. Production systems rarely operate exactly as planned; machine failures, unexpected arrival of urgent orders, and customer cancellations of ongoing operations are all possible. This embodiment provides a way to handle these exceptional situations, ensuring that the scheduling system remains stable and continues to operate effectively in the event of problems.

[0124] Anomaly handling mechanisms can be implemented individually or in combination, depending on the needs of the production environment.

[0125] (1) Machine troubleshooting:

[0126] When a machine malfunction is detected, the response is to set the processing time of all currently running and pending operations on the malfunctioning machine to zero. This mechanism handles situations where a machine is unavailable due to malfunction, maintenance, or other issues. The monitoring system continuously tracks the status of each machine. When a malfunction is detected, such as through sensor data, operator reports, or maintenance alarms, all affected operations are immediately identified. This includes operations currently running on the malfunctioning machine, as well as any operations scheduled or waiting to be processed on that machine in the future. For currently running operations, setting their processing time to zero effectively cancels any remaining processing. For operations scheduled or waiting to be processed on the malfunctioning machine, setting their processing time to zero removes them from consideration for that machine, indicating that they are no longer a viable option for processing. After this adjustment, the affected operations may need to be reassigned to other eligible machines (if any). The reinforcement learning model will discover in subsequent decision cycles that these operations have zero processing time on the malfunctioning machine, and therefore will only consider other machines in the set of available machines. If no alternative machine is available, the operations may remain pending until the machine is repaired or other arrangements are made.

[0127] For example, consider a factory where welding machine M2 malfunctions while processing operation A2. At the time of the malfunction, operation A2 has 15 minutes remaining. The method immediately sets the remaining processing time of A2 on M2 to zero. Furthermore, any other operations scheduled on M2, such as B2 waiting to start, also have their processing time on M2 set to zero. In subsequent decision cycles, the reinforcement learning model will only consider other alternative machines for A2 and B2. If no other welding machines are available, these operations will wait until M2 is repaired or an alternative arrangement is made.

[0128] (2) Emergency order processing:

[0129] When an urgent order is detected, the new job is added to a buffer queue to await loading in response. This mechanism handles situations where high-priority jobs arrive unexpectedly and need to be incorporated into the production schedule as quickly as possible. The monitoring system detects when an urgent order is placed, which can be triggered by manual input, automatic alerts from the ordering system, or priority flags associated with certain customers or products. Upon detecting such an event, the urgent job is not immediately inserted into the proactive workflow, as this could disrupt ongoing scheduling decisions and potentially overload the system. Instead, the urgent job is placed in a dedicated buffer queue for urgent orders. Jobs in the urgent buffer queue are prioritized when capacity is available. When workflow capacity becomes available, such as when an existing job completes and is removed, the urgent buffer queue is checked first, and waiting urgent jobs are loaded before regular jobs are loaded from the standard buffer queue. This ensures that urgent orders are processed as quickly as possible without disrupting the ongoing scheduling process.

[0130] For example, a factory receives an urgent order from a key customer, requiring the delivery of a part within 24 hours. The monitoring system detects this urgent order and places it in an urgent buffer queue. The active workflow is currently full, with 100 processes. As processes complete and capacity becomes available—for example, when a job completes and its 5 processes are removed—the method first checks the urgent buffer queue, finds the urgent job, and loads it into the workflow before any regular waiting jobs. The urgent job's processes become available for scheduling in the next decision cycle.

[0131] (3) Job cancellation processing:

[0132] When job cancellation is detected, the system responds by marking all incomplete operations within the job's operation subset as completed in the Directed Acyclic Graph (DAG). This mechanism handles situations where customers cancel orders, production schedules change, or jobs are no longer needed. The monitoring system detects job cancellations through the order management system, manual input, or automated schedule updates. Upon detection of cancellation, all incomplete operations belonging to that job are identified. These operations are then marked as completed in the DAG, effectively removing them from the active workflow. Marking cancelled operations as completed, rather than simply deleting them, preserves the integrity of the DAG structure.

[0133] For example, a customer cancels an order for a custom-made part in production. Job C, comprising operations C1 and C2, is canceled. At the time of cancellation, C1 is completed, but C2 has not yet started. This method marks C2 as completed in the DAG. Any operations in other jobs that depend on C2, such as assembly operations requiring the custom part, now see that C2 is completed and may become executable if their other dependencies are satisfied. This correctly reflects that the cancellation of job C eliminates the need for its remaining operations to wait.

[0134] First, the exception handling mechanism enables the scheduling system to remain stable and continue operating in the event of unexpected events, avoiding catastrophic failures or system crashes. Second, by handling exceptions, the need for a complete rescheduling from scratch is avoided, which would be computationally costly and time-consuming. Furthermore, the exception handling mechanism is designed to integrate seamlessly with existing DAGs and state representations; setting processing time to zero, adding to a cache queue, and marking processes as completed all maintain the integrity of the scheduling framework.

[0135] In one embodiment, the reinforcement learning model makes decisions for each machine at preset time slot intervals. In each time slot, the machine currently needing to make a decision selects the task to be executed from the set of currently available tasks. The decision space of the reinforcement learning model includes idle actions, which indicate that the machine currently needing to make a decision should not execute a task in the current time slot. The reinforcement learning model operates according to a structured time frame that determines when and how decisions are made. This frame ensures that the scheduling system maintains regular and predictable decision points, while providing flexibility to handle situations where there are no suitable tasks to execute immediately.

[0136] For example, consider a factory with a time slot period of five minutes. At the start of a specific time slot, three machines M1, M2, and M3 need to make a decision: M1 has just completed its previous operation and is idle; M2 has been idle for the past two time slots; M3 is currently busy and needs three more time slots before needing to make a decision. The set of currently executable operations includes B2, C1, and D3. For machine M1, the model considers its options: it can execute C1 or D3, or it can choose to be idle. For machine M2, the options are B2, D3, or idle. Assuming the model already knows that operation B2 is critical because it unlocks two downstream operations, it decides that M2 should execute B2. For M1, the model finds that executing C1 is now possible, and confirms that the high-priority operation A4 will become executable in the next time slot and is a good fit for M1. Therefore, the model chooses the idle operation for M1, keeping it idle for the more important operation that is about to arrive. M2 executes B2, and M1 remains idle during this time slot.

[0137] In the next time slot, A4 becomes executable as predicted, and M1 is immediately available to begin processing. The strategic idle period pays off, as M1 is ready when high-priority operations become available.

[0138] First, the slotted decision-making framework with idle actions provides a structured and predictable decision-making rhythm, integrating well with reinforcement learning paradigms that typically operate on discrete-time steps. Second, it improves efficiency by focusing computational resources where they are most needed, making decisions only for the machines that require them. Furthermore, idle actions add flexibility to the model's policy.

[0139] In one embodiment, when the machine currently requiring a decision is selected for an idle action, the machine does not perform any operations in the current time slot and re-engages in the decision-making process in the next time slot. This embodiment further specifies the behavior of the machines when an idle action is selected and how they re-enter the decision-making process in subsequent time slots. This clarification ensures that the scheduling system appropriately handles idle periods and maintains continuous operation over time.

[0140] Specifically, when the reinforcement learning model selects an idle action for a machine that currently needs to make a decision, that machine does not perform any operations during the current time slot. This means that the machine remains inactive for the entire duration of the time slot, neither starting any new work nor making progress on any operations. The machine simply waits, consuming no resources and producing no output during this period. The decision to idle the machine is not permanent. Rather, it is a temporary state applicable only to the current time slot. At the end of the current time slot and the beginning of the next time slot, the machine automatically re-engages in the decision-making process. It is again considered a machine that needs to make a decision, and the reinforcement learning model will again evaluate whether to assign it an operation or leave it idle for another time slot. This cycle can continue to repeat, and the machine may be idled for multiple consecutive time slots if the model consistently determines that idleness is the best option. In each time slot, a new decision is made based on the current state information and the current set of executable operations. As the production environment changes, new operations become executable, the machine completes other work, the job progresses, and the model's evaluation of whether idleness is beneficial may change accordingly.

[0141] Consider a factory with a five-minute time slot period. Machine M1 is idle at the start of time slot t. A reinforcement learning model evaluates the current situation and decides to choose the idle action for M1. For the entire five-minute duration of time slot t, M1 remains idle; it does not initiate any operation. When time slot t ends and time slot t+1 begins, M1 automatically becomes the machine requiring a decision again. The model re-examines the current state information and the set of executable operations. Perhaps in this new time slot, a high-priority operation that is very suitable for M1 has become available. The model now chooses to execute that operation, and M1 begins working. Alternatively, if the situation remains unchanged and idleness is still the best option, the model may again choose the idle action, and M1 will also remain idle in time slot t+1. This situation may persist for several time slots until the conditions change and the model determines that initiating an operation is more preferable.

[0142] In one embodiment, the workflow has a preset upper limit on the number of processes. When a new job arrives and the number of processes in the workflow is less than the upper limit, the new job is added to the workflow. When a new job arrives and the number of processes in the workflow is not less than the upper limit, the new job is added to a buffer queue. This method further manages the flow of new jobs into the production system by enforcing capacity limits on the workflow. This capacity management ensures that the scheduling system operates within manageable limits, preventing overload and maintaining scheduling quality.

[0143] A workflow has a preset upper limit on the number of operations it can contain at any given time. This upper limit can be a configurable parameter, determined by various factors such as the physical capacity of the production floor, the computing power of the scheduling system, the response speed required by the scheduling algorithm, or the management policy of work-in-process inventory. For example, a factory might set a limit of 200 operations to ensure that the scheduling model can make decisions quickly and that the production floor does not become congested due to too many waiting jobs. When a new job arrives, the current occupancy of the workflow is first checked by examining how many operations are currently in place. This includes all operations of all jobs in the workflow, those waiting to start, those currently executing, and those scheduled but not yet started. The method then compares the current number to the preset upper limit. If the number of operations in the workflow is less than the upper limit, it means that adding a new job will not exceed the capacity constraint, and the new job is added directly to the workflow. A subset of the job's operations is incorporated into the Directed Acyclic Graph (DAG), the current state information is updated to include the new operation, and the job becomes available for scheduling in subsequent decision cycles. This on-the-spot integration ensures that new jobs enter the production flow as quickly as possible when capacity allows. If the number of processes in the workflow is not less than the upper limit (meaning it is equal to or greater than the limit), new jobs will not be added directly to the workflow. Instead, new jobs will be added to a cache queue to wait for later loading. The cache queue serves as a temporary storage area for jobs that arrive due to capacity limitations but cannot yet be accommodated. Jobs in the cache queue wait until capacity becomes available, and when existing jobs are completed and removed from the workflow, they can then be loaded from the queue into the active workflow.

[0144] Consider a manufacturing facility with its workflow capacity limited to 100 operations. At one point, the workflow contains 85 operations across various jobs at different completion stages. A new job arrives, job X, consisting of 5 operations. The process checks the current occupancy: there are already 85 operations in the workflow. Adding the 5 operations to job X will bring the total to 90 operations, still less than the 100-operation capacity limit. Therefore, the workflow has available capacity, and job X is added directly to the workflow. Its operations become part of the current state and will be considered in future scheduling decisions. Later, another new job arrives, job Y, consisting of 8 operations. At this point, the workflow contains 95 operations. Adding the 8 operations to job Y will bring the total to 103 operations, exceeding the 100-operation capacity limit. Therefore, the workflow is full, and job Y cannot be added immediately. Instead, it is placed in a buffer queue, where it waits. Over time, existing jobs complete, and their operations are removed from the workflow. When workflow occupancy drops below capacity limits, for example when enough jobs complete to reduce the total number of operations to 92, the method automatically checks the cache queue and loads waiting jobs. Job Y, which has been waiting in the queue, is now added to the active workflow, and its operations become available for scheduling.

[0145] The workflow capacity management method described above offers significant advantages. First, it prevents the scheduling system from becoming overloaded by too many pending tasks, which could reduce decision quality or increase computational overhead. Furthermore, by enforcing capacity limits, it helps control work-in-process inventory on the production floor, reducing congestion and improving flow. Additionally, the buffer queue provides a buffer to handle periods of high demand; when jobs arrive faster than can be processed, they wait in the queue instead of causing system overload. Moreover, automatic loading from the buffer queue when capacity becomes available ensures that waiting jobs are not forgotten and are eventually processed.

[0146] In one embodiment, the method further includes: when a new job is added to the workflow, identifying processes in the process subset of the new job that are of the same type as processes in the process subset of existing jobs; merging the processes in the process subset of the new job and the processes in the process subset of existing jobs that are of the same type into a single node in the DAG, and transferring the original dependencies of the processes in the process subset of the new job and the processes in the process subset of existing jobs that are of the same type to the merged node. In this embodiment, the ability to merge processes of the same type is further extended to handle the dynamic arrival of new jobs. The aforementioned dynamic operation merging mechanism identifies and integrates processes of the same type in different jobs. This embodiment further extends this capability to handle the arrival of new jobs.

[0147] When a new job is added to the workflow, the method immediately checks its subset of operations to identify if any operations are of the same type as those already existing in the workflow. This identification process compares the type of each operation in the new job with the types of all operations in the current workflow. Operation types are determined by the nature of the required processing, such as heat treatment, painting, curing, machining, etc. For example, suppose the workflow already contains several jobs that include heat treatment operations: job A has operation A3, job C has operation C2, and job E has operation E4. These existing heat treatment operations may have been merged into a heat treatment batch node in the DAG, or they may still be separate nodes, depending on when they entered the workflow and whether a merge had occurred previously. When a new job F arrives and its subset of operations includes F2, the method identifies that this new operation is of the same type as an existing heat treatment operation. Once this identification is made, the method merges all operations of the same type, whether newly arrived or existing, into a node in the DAG.

[0148] Consider a factory workflow that already contains three jobs with heat treatment operations: Job A: Operation A3, dependent on A2; Job C: Operation C2, dependent on C1; Job E: Operation E4, dependent on E3. These three heat treatment operations have been merged into a node in the Directed Acyclic Graph (DAG) called "Heat Treatment Batch 1". This merge node has three prerequisites: A2, C1, and E3 must all be completed before the batch begins. After the batch is completed, operations A4, C3, and E5 become available to begin. Now, a new job G arrives. Its subset of operations includes G3, dependent on G2. The method identifies that G3 is the same type of operation as the existing operation in the Heat Treatment Batch 1 node. It then creates a new merge node "Heat Treatment Batch 2", merging all four heat treatment operations: the three existing operations from jobs A, C, and E, plus the new operation from job G. Dependencies are transferred: the new merge node now depends on A2, C1, E3, and G2. After the batch is completed, operations A4, C3, E5, and G4 all become available to begin. The DAG was updated to reflect this new merge node, and the standalone operating node was removed from further consideration.

[0149] Dynamic operation merging upon the arrival of new jobs offers significant advantages. First, it ensures that merging opportunities are not missed; once a new job enters the system, the potential of any batch is immediately identified and utilized. Furthermore, by maintaining the DAG as an accurate and up-to-date representation of the production workflow, the merge node reflects the current set of jobs in the system. Additionally, by transferring all dependencies to the merge node, it preserves the correctness of the scheduling logic, ensuring that all preconditions are met before batch processing can begin, and that all subsequent operations correctly wait for batch completion.

[0150] In one embodiment, the reinforcement learning model is trained using the policy gradient method; the DAG is described in at least one of the following ways: a visual graph; an adjacency list; JSON / YAML format; or the Cypher query language. The reinforcement learning model is trained using the policy gradient method. These various description formats provide flexibility for different implementation scenarios; visual graphs are used for design and documentation, adjacency lists for algorithm efficiency, JSON / YAML for system integration, and Cypher for graph database management.

[0151] Figure 4 This is a schematic diagram of a Directed Acyclic Graph (DAG) used to describe process dependencies in an embodiment of the present invention. Figure 4 As shown, DAG provides a unified framework for describing complex dependencies between processes across multiple jobs, demonstrating structures such as multi-source, branching, and merging that traditional matrix representations cannot capture. Figure 4 The DAG shown is divided into two parts. In the first part, process O... 11 Pointing to O 12 , indicating O 12 Only in O 11 Start after completion. 12 and O 13 Both point to O 14 This demonstrates a merge structure where O 14 O is needed 12 and O 13 Everything must be completed before we can begin. 14 and O 21 Then it points to a representation of O. 15 and O 22 The composite node demonstrates that processes from different jobs share a common dependency. This composite node then points to O. 16 And point to O 23 This illustrates a branching structure where the completion of a combining node enables two downstream processes to proceed. Figure 4 In the second part, O 31 and O 32 Both point to O 33 , representing another confluence structure, where O 33 O is needed 31 and O 32 All done. 33 Then point to O 34 After merging, they form a linear sequence. This DAG example illustrates several key characteristics of complex dependencies: multiple source operations, merging points, cross-job dependencies, and branching structures. Each job in the workflow contains a subset of procedures within this global DAG. For example, the procedure subset for job 1 includes node O. 11 O 12 O13 O 14 O 15 O 16 and the edges connecting them; the process subset of task 2 includes O 21 O 22 O 23 The process subset of assignment 3 includes O 31 O 32 O 33 O 34 These subsets can intersect at shared nodes, enabling the representation of cross-job dependencies and parallel processing opportunities.

[0152] Figure 5 This is a schematic diagram illustrating the reinforcement learning principle in an embodiment of the present invention. Figure 5 As shown, the reinforcement learning framework establishes an interactive loop between agent 10 and its environment 20. This infrastructure enables the system to learn optimal scheduling strategies through experience. At the center is agent 10, within which is the reinforcement learning model responsible for making scheduling decisions. Surrounding the agent is the environment 20, representing the production workflow that includes all jobs, machines, processes, and their current states.

[0153] The interaction process is as follows: At each time step t, the environment 20 provides the agent 10 with a state representation St. This state includes all information about the current workflow, such as which machines are idle or busy, which processes are in progress, which processes have been completed, and which jobs are waiting. The state serves as the perceptual input that informs the agent's decision-making. Based on this state St, the agent 10 selects an action A from its available action space. t In a scheduling context, actions include selecting to execute a specific process on a specific machine, or selecting to leave a machine idle. Actions represent an agent's decision about what should happen next in the production workflow. Agent 10 performs this action A. t The message is relayed back to environment 20, and environment 20 then executes the decision. As a result of taking the action, environment 20 transitions to a new state S. t+1 The machine begins processing, the process progresses, and the workflow evolves. Simultaneously, environment 20 generates a reward R. t+1 The agent evaluates the quality of the actions taken. Positive rewards reinforce their decisions, making similar actions more likely to occur in similar future states. Negative rewards inhibit related decision patterns. The agent's goal is to learn a policy: a mapping from states to actions that maximizes the cumulative reward over time. Then, the cycle repeats: new state S t+1 Provided to agent 10, agent 10 selects the next action A. t+1And so on. Through extensive interactions, Agent 10 gradually refines its strategies, learning complex scheduling tactics to balance short-term efficiency with long-term optimization. In the context of flexible job shop scheduling, the reinforcement learning framework enables the system to discover subtle strategies that are difficult to encode in rule-based or heuristic methods. Agent 10 learns to recognize patterns through direct interaction with the production environment, such as when to prioritize certain operations, when to strategically idle machines, and how to balance competing objectives.

[0154] Figure 6 This is a schematic diagram of the process index matrix in an embodiment of the present invention. Figure 6 As shown, the process index matrix records the process identifiers at each position in the production plan for each machine. This matrix has dimensions N×M, where M represents the number of machines in the workflow, N represents the workflow capacity, and the maximum number of processes that each machine can queue or plan. Figure 6 In the example matrix shown, the rows of the matrix are P1, P2, ..., P... N Identifier, where P n This represents the nth position in the workflow, with columns M1, M2, ..., M... M Each machine is identified by an index. The matrix indexes M machines from 1 to M, each machine having a production plan that can accommodate up to N operations. The element in the nth row and mth column of the matrix represents the operation scheduled at that position in the production plan for machine m. Each operation is identified by a tuple (j, k), where j represents the job index and k represents the operation index within that job. For example, the entry (1, 2) represents the second operation of job 1. The tuple (0, 0) is used as a placeholder to indicate that the operation cannot be processed on that machine, or that the machine is unavailable at that position. This matrix provides several important functions. First, it establishes a clear mapping between operations and their assigned positions in the queue of each machine. Second, it enables the system to track the planned operation sequence for each machine, which is crucial for understanding future workloads and potential bottlenecks. Third, the fixed-size matrix format with zero padding allows reinforcement learning models to handle workflow states of varying sizes using a consistent input structure. The operation index matrix works in conjunction with other data structures shown in subsequent figures to provide a complete picture of the workflow states. While this matrix indicates what processes are planned for each machine, the process processing time matrix indicates how long each planned process will take, and the allocation vector and remaining time vector indicate what is happening in real time.

[0155] Figure 7 This is a schematic diagram of the process time matrix in an embodiment of the present invention. Figure 7As shown, the process processing time matrix records the processing time required for each process on each machine. This matrix shares the same dimension N×M as the process index matrix, where M is the number of machines and N is the workflow capacity. The rows of the matrix are numbered P1, P2, ..., P... N Identifier, where P n This represents the nth position in the workflow, with columns M1, M2, ..., M... M Each step is identified by an identifier, corresponding to a specific machine. Each element in the matrix corresponds positionally to the same element in the step index matrix. For each step identified in the step index matrix, the corresponding element in the step processing time matrix indicates how long that step will take to complete on its assigned machine. For example, if the step index matrix shows that step (1,2) is scheduled in row 3, column 2, then the element in row 3, column 2 of the processing time matrix will contain the processing time for that step on machine 2. Zero values ​​in this matrix have special meaning. They may indicate that the corresponding step cannot be processed on a particular machine, or that no step is scheduled for that position. This zero-padding clearly distinguishes between unavailable machines and empty queue positions while maintaining a fixed-size matrix structure. The processing time matrix provides necessary information for several aspects of the scheduling approach. It can predict step completion times, which is crucial for determining when machines become available and when dependent steps can begin. It also allows reinforcement learning models to evaluate the temporal impact of different machine allocation decisions, suggesting that selecting faster machines for critical steps may be beneficial, while reserving slower machines for less urgent tasks. The same process may take different times on different machines, reflecting differences in machine speed, capacity, or tools. This flexibility is a key feature of flexible job shop scheduling, and the processing time matrix captures these differences in a format that reinforcement learning models can easily use.

[0156] Figure 8 This is a schematic diagram of the process allocation vector in an embodiment of the present invention. Figure 8 As shown, the operation assignment vector records the operations currently being executed on each machine at a given time. This is a one-dimensional vector of length M, where M is the number of machines. Each element corresponds to a specific machine, indicating the operation currently in progress on that machine. Figure 8In the example vectors shown, each entry contains a tuple (j,k) identifying the currently executing operation, where j is the job index and k is the operation index within that job. For example, the entry (2,1) would indicate that machine m is currently executing the first operation of job 2. The tuple (0,0) indicates that the machine is currently idle and not processing any operations. Unlike traditional scheduling representations that assume each machine can only process one operation at a time, operation allocation vectors are designed to adapt to more complex scenarios. They allow the same job index to appear multiple times simultaneously on different machines. This capability is crucial for representing jobs with multiple sources and branches; for example, different parts of the same car body may be processed in parallel on different machines, and the allocation vector accurately reflects multiple operations from the same job being executed concurrently. Operation allocation vectors provide a real-time snapshot of proactive processing across the entire production floor. This information is essential for determining when machines are available, identifying bottlenecks, and making informed scheduling decisions about what to start next.

[0157] Figure 9 This is a schematic diagram of the remaining time vector of a process in an embodiment of the present invention. Figure 9 As shown, the process remaining time vector records the remaining processing time of the currently executing process on each machine. This vector has a length M and corresponds positionally to the process assignment vector. For each machine, this value represents how much time is left to complete the current process. Figure 9 In the example vector shown, each entry contains a numerical value representing the number of minutes remaining. For idle machines, the remaining time is zero. For machines actively processing, the value reflects the remaining time based on the total processing time and the time already run for the process. The remaining time vector provides crucial timing awareness for scheduling systems. By knowing exactly when each machine will be idle, the system can predict future availability and make smarter decisions about which processes to start now and which to reserve for upcoming opportunities. For example, if a high-priority process will become available in 10 minutes, and a suitable machine will become idle in 8 minutes, the system might choose to temporarily idle that machine rather than start a less important process that would take it longer. This vector is updated in real time as time progresses and as processes complete or are interrupted. It works in conjunction with the allocation vector to provide a complete picture of what is happening now and what will happen in the near future.

[0158] Figure 10 This is a schematic diagram of the process dependency matrix in an embodiment of the present invention. Figure 10As shown, this matrix records the dependencies between processes in the workflow in tabular form. The process dependency matrix is ​​an N×N matrix, where N is the workflow capacity, and the rows and columns of the matrix correspond to each process. The element in the u-th row and v-th column indicates whether the start of process u depends on the completion of process v: a value of 1 indicates that process v must be completed before process u begins, and a value of 0 indicates no dependency. The diagonal elements are usually zero because processes do not depend on themselves. Figure 10 In the matrix shown, rows and columns correspond to O. 11 To O 34 Each process. For example, it can be seen that: process O 12 Depends on process O 11 (O) 12 Line, O 11 Column value is 1); Process O 13 Depends on process O 12 (O) 13 Line, O 12 Column value is 1); Process O 14 Depends on process O 12 and O 13 (O) 14 Line, O 12 Column and O 13 Column value is 1); Process O 15 Depends on process O 14 (O) 15 Line, O 14 Column value is 1); Process O 16 Depends on process O 15 (O) 16 Line, O 15 Column value is 1); Process O 23 Depends on process O 22 (O) 23 Line, O 22 Column value is 1); Process O 33 Depends on process O 32 (O) 33 Line, O 32 Column value is 1); Process O 34 Depends on process O 33 (O) 34 Line, O 33 (Column values ​​are 1); other unlisted dependencies can be determined in a similar way. This matrix encodes the complete dependency structure of the workflow in a format that reinforcement learning models can efficiently process. Reading a row identifies all prerequisite processes that must be completed before the current process can begin; reading a column identifies all subsequent processes that depend on the completion of this process. This representation allows reinforcement learning models to quickly query the prerequisite dependencies for each process without needing to parse the graph structure in real time.

[0159] Figure 11 This is a schematic diagram of a process dependency matrix segment of a streaming workflow in an embodiment of the present invention. Figure 11 This illustrates how parallel processes from different jobs are integrated when a new job is added to the streaming workflow. When a new job arrives, processes of the same type from different jobs may be merged into a single node in the dependency matrix. For example, processes O from different jobs... 15 and O 22 It is integrated and represented as a single node P in the process dependency matrix of the streaming workflow. i+4 The dependency was correctly transferred, so that Pᵢ₊4 initially depends on its predecessor P. i+3 and P i+6 This integration mechanism enables efficient batch processing of the same process across different jobs, improving equipment utilization and reducing total production time.

[0160] Figure 12 This is a schematic diagram illustrating action selection in an embodiment of the present invention. Figure 12 As shown, the action selection process defines how the reinforcement learning model chooses what to do at each decision time. The action space is shown as ranging from 0 to J. N The range of J, where J N This indicates the maximum number of jobs the workflow can support. In this action space, a value of 0 represents a special idle action. When the model selects 0 for a machine, it means that the machine should remain idle in the current time slot and not execute any actions. This action might be selected when there are no suitable available actions for the machine, or when strategic waiting is beneficial, such as reserving capacity for high-priority actions that will soon become available. (From 1 to J) N The value represents the job index. When the model selects a value j within this range, it means that the next action to be executed on the current machine should be taken from job j. Specifically, the model selects the next executable action suitable for this machine from a subset of the actions in that job. The action selection process is invoked for each machine that needs to make a decision in each time slot. The model considers the current state and generates a probability distribution over possible actions or a specific action selection. This decision not only determines which job to process, but also implicitly determines which specific action within that job, based on the currently executable actions suitable for the machine.

[0161] Figure 13 This is a schematic diagram of a policy network in an embodiment of the present invention. Figure 13As shown, the policy network 40 is a neural network architecture that implements the decision-making function of a reinforcement learning model. This network takes the current state 30 as input and produces a probability distribution 50 over possible actions or a specific action recommendation. The input layer of the policy network 40 receives current state information, including all five data structures described above (operation index matrix, operation processing time matrix, operation assignment vector, operation remaining time vector, and operation dependency matrix). These inputs are concatenated or otherwise combined to form a comprehensive representation of the workflow state. The network processes this input through multiple hidden layers to extract relevant features and patterns. These hidden layers may include convolutional layers for processing matrix inputs, fully connected layers for integrating information across different data structures, and specialized layers for processing graph-structured data from the dependency matrix. The output layer produces action decisions. In a stochastic policy implementation, the output is a probability distribution over all possible actions, with higher probabilities assigned to actions the model believes will yield better rewards. In a deterministic policy implementation, the output is a specific action selection. The policy network is trained empirically, with its weights adjusted over time to make better decisions. Training typically employs policy gradient methods, such as REINFORCE or the Actor-Critic algorithm, which update network parameters in directions that increase the probability of actions leading to higher rewards. Through this training process, the network gradually learns the optimal or near-optimal scheduling policy for the target production environment.

[0162] To better illustrate this, the specific implementation process of the embodiments of the present invention in the training phase and the inference phase are described below.

[0163] I. Training Phase

[0164] This example demonstrates how to train a reinforcement learning model using training sample jobs.

[0165] Consider a small simulation workshop equipped with three machines: a cutting machine M1, a welding machine M2, and a heat treatment furnace M3. The training dataset contains three types of training sample jobs: a car body manufacturing training sample job requires cutting, welding, and heat treatment in sequence, denoted as operations A1, A2, and A3 of training sample job A; a car door manufacturing training sample job requires cutting and welding in sequence, denoted as operations B1 and B2 of training sample job B; and an engine bracket manufacturing training sample job requires cutting and heat treatment in sequence, denoted as operations C1 and C2 of training sample job C. The dependencies between the operations of all training sample jobs are described by a Directed Acyclic Graph (DAG). The available machines for each operation are as follows: cutting can only be performed by M1, welding can only be performed by M2, and heat treatment can only be performed by M3. The processing times for each operation are: cutting 30 minutes, welding 45 minutes, and heat treatment 60 minutes. The workflow has a preset maximum of 10 operations and a time slot period of 5 minutes.

[0166] At the initial time t=0, the workflow is empty. Training sample job A arrives first and is added to the workflow since it is not full. Its process subsets A1, A2, and A3 are incorporated into the DAG, and the current state information is updated accordingly. At t=5, training sample job B arrives and is added to the workflow. At t=10, training sample job C arrives and is added to the workflow. At this point, based on the current state information and the DAG, the set of currently executable processes is determined. Since A1, B1, and C1 have no preceding dependent processes, all three are executable, and the current executable set is {A1, B1, C1}.

[0167] At time slot t=10, the reinforcement learning model makes decisions for each machine according to a preset time slot period. The machines currently requiring decision-making are the idle M1, M2, and M3. For the cutting machine M1, the model inputs its current state information and the executable set, deciding that the current operation to be executed is B1. It assigns M1 to B1 from the set of available machines {M1} corresponding to B1, and M1 begins executing B1, expected to complete in 30 minutes. For the welding machine M2 and the heat treatment furnace M3, there is no suitable welding operation for M2 in the current executable set because A2 depends on A1, and B2 depends on B1, neither of which is satisfied. Similarly, there is no suitable heat treatment operation for M3 because A3 depends on A2, and C2 depends on C1, neither of which is satisfied. Therefore, the model selects an idle action for M2 and M3, meaning they do not execute any operations in the current time slot.

[0168] At t=30, M1 completes B1, and the state of B1 is updated to "completed". Based on the updated current state information and the DAG, the executable set is redefined. It is found that B2's prerequisite B1 has been completed, so B2 becomes executable, while A1 and C1 remain executable. The executable set is now updated to {A1, C1, B2}. The model makes a decision for M1, which is now idle, choosing A1 and executing it on M1.

[0169] At t=40, M2 and M3 are idle, while M1 is executing A1, expected to complete at t=60. The current executable set is {C1, B2}. The model makes a decision for M2, selecting B2 and executing it on M2, and selecting an idle action for M3.

[0170] At t=60, M1 completes A1, and A2 becomes executable. However, the only available machine for A2 is M2, which is currently executing B2 and is expected to complete at t=85. Therefore, the model selects an idle action for M1, waiting for M2 to become available.

[0171] At t=85, M2 completes B2, at which point all operations (B1 and B2) of the training sample job B have been completed. The system detects the completion of the training sample job B and calculates a reward value based on its completion information: the number of operations is 2, and the actual processing time from the first operation B1 (t=10) to the completion of the last operation B2 (t=85) is 75 minutes. The reward value is calculated as 2 / 75≈0.0267, and the parameters of the reinforcement learning model are updated based on this reward value.

[0172] At t=100, C1 completes, and C2 becomes executable. At this point, the system recognizes that C2 and A3 are the same type of heat treatment process, triggering the same-type process merging mechanism. In the DAG, C2 and A3 are merged into a single heat treatment merge node, and the original dependencies between C2 and A3 are transferred to this merge node. Since the prerequisites A2 and C1 for the merge node have been satisfied, the merge node becomes executable. The model, M3, decides to execute this merge node. M3 simultaneously processes the heat treatment processes of training sample jobs A and C. The heat treatment process takes 60 minutes and is expected to complete at t=160.

[0173] At t=130, M2 completes A2. At t=160, M3 completes the heat treatment merging node. At this point, all processes of training sample jobs A and C have been completed. The system calculates reward values ​​for training sample jobs A and C respectively: Job A has 3 processes, and the actual processing time from A1 (t=30) to A3 (t=160) is 130 minutes, with a reward value of 3 / 130≈0.0231; Job C has 2 processes, and the actual processing time from C1 (t=40) to C2 (t=160) is 120 minutes, with a reward value of 2 / 120≈0.0167. The parameters of the reinforcement learning model are further updated based on these reward values.

[0174] Through repeated iterations with a large number of training sample jobs, the model gradually learns optimized scheduling strategies. Once training is complete, the model parameters are fixed, and it can be deployed to a real production environment.

[0175] II. Reasoning Stage

[0176] This example demonstrates the application of a trained reinforcement learning model in real-world production. Assuming the model is already trained and its parameters are fixed, no reward value is calculated or the model parameters are updated during the actual scheduling process.

[0177] Assume the workshop configuration, job type, machine capacity, etc., are the same as in the training phase. A batch of actual production jobs arrives: at initial time t=0, the workflow is empty. Actual job A arrives first and is added to the workflow because it is not full. At t=5, actual job B arrives. At t=10, actual job C arrives. The subsequent decision-making and execution process is the same as in the training phase: the model selects the operation to be executed for the machine requiring decision-making in each time slot based on the current state information and assigns the machine accordingly. The execution order of operations, the merging mechanism of operations of the same type, etc., are consistent with the training phase. The main difference from the training phase is that when an actual job is completed, the system only records its completion status to update the workflow's operation occupancy; it does not calculate reward values ​​or update model parameters. The model only makes decisions based on the learned fixed strategies and no longer learns from actual production. At t=120, a new actual job D arrives. At this time, the number of operations in the workflow is 8, which is lower than the upper limit of 10, so job D is directly added to the workflow. At t=200, another new actual job E arrives. At this point, the number of processes in the workflow has reached the upper limit of 10, so job E is added to the cache queue, waiting to be loaded when capacity becomes available.

[0178] Through the above process, the trained reinforcement learning model can continuously and efficiently schedule actual production operations, while avoiding unnecessary model updates in the production environment.

[0179] As shown in the above implementation process, the embodiments of the present invention can effectively handle complex job dependencies described by DAG, use reinforcement learning models to dynamically determine the processes to be executed and the machines to be allocated in each time slot, support the merging of the same type of processes to make full use of equipment capacity, calculate reward values ​​based on job completion information to continuously optimize scheduling strategies, and adapt to the dynamic arrival of new jobs through capacity management and cache queue mechanisms.

[0180] The present invention also proposes an electronic device with a processor-memory architecture. Figure 14 This is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 14 As shown, the electronic device 800 includes a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program, it implements any of the aforementioned PLC behavior simulation methods. The memory 802 can be implemented as various storage media, such as electrically erasable programmable read-only memory, flash memory, or programmable read-only memory. The processor 801 can be implemented as one or more central processing units (CPUs) or one or more field-programmable gate arrays (FPGAs), wherein the FPGA integrates one or more CPU cores. Specifically, the CPU or CPU core can be implemented as a CPU, MCU, or DSP, etc.

[0181] It should be noted that not all steps and modules in the above flowchart and structure diagram are necessary and can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The division of modules is for descriptive purposes only; in actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can be implemented by the same module. These modules can reside in the same device or in different devices.

[0182] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuitry or logic devices to perform a specific operation. A hardware module may also include programmable logic devices or circuitry temporarily configured by software to perform a specific operation. The specific implementation method chosen—whether mechanical, using dedicated permanent circuitry, or using temporarily configured circuitry—can be determined based on cost and time considerations.

[0183] The present invention also provides a machine-readable storage medium storing instructions for causing a machine to perform the methods described herein. Specifically, a system or apparatus equipped with a storage medium storing software program code implementing the functions of any of the embodiments described above can be provided, and a computer of the system or apparatus can read and execute the program code stored in the storage medium. Furthermore, based on the instructions in the program code, an operating system or the like running on the computer can perform some or all of the actual operations. The program code read from the storage medium can also be written to a memory on an expansion board in the computer or a memory in an expansion unit connected to the computer. A control unit installed on the expansion board or expansion unit can then perform some or all of the actual operations based on the instructions in the program code, thereby implementing the functions of any of the embodiments described above. Storage media for providing the program code include floppy disks, hard disks, magneto-optical disks, optical disks, magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer or the cloud via a communication network.

[0184] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A flexible job shop scheduling method, characterized in that, include: Obtain the current status information of the (101) workflow, which includes multiple jobs and multiple machines, wherein the dependencies between the operations of the multiple jobs are uniformly described by a directed acyclic graph (DAG), each job includes a subset of operations in the DAG, each operation corresponds to at least one optional machine, and each optional machine is selected from the multiple machines. Based on the current state information and the DAG, determine (102) the set of currently executable processes to be processed; The current state information and the set of processes to be processed are input into the reinforcement learning model (103) so that the reinforcement learning model can determine the process to be executed in this process from the set of processes to be processed, and allocate a machine to the process from at least one optional machine corresponding to the process to be executed in this process. as well as The process to be performed is executed (104) on the assigned machine.

2. The method according to claim 1, characterized in that, It also includes the training process of the reinforcement learning model; During the training process, for each training sample task, when it is detected that all the procedures included in the training sample task have been completed, a reward value is calculated based on the completion information of the training sample task, and the parameters of the reinforcement learning model are updated based on the reward value.

3. The method according to claim 2, characterized in that, The calculation of the reward value based on the completion information of the training sample job includes: Obtain the number of processes contained in the process subset of the training sample job where all processes have been completed; Obtain the actual processing time of the training sample job from the start of the first process to the completion of the last process; The reward value is calculated based on the number of processes and the actual processing time.

4. The method according to claim 3, characterized in that, The training process also includes: Obtain the deadline for the training sample job after all the above procedures have been completed; The reward value is adjusted based on the deadline, wherein completing the task ahead of time results in a positive reward, completing it on time results in a zero reward, and completing it after the deadline results in a negative reward.

5. The method according to claim 3, characterized in that, The calculation of the reward value based on the number of processes and the actual processing time includes: for training sample jobs with the same number of processes, the shorter the actual processing time, the higher the reward.

6. The method according to claim 1, characterized in that, The acquisition of the current status information of workflow (101) includes at least one of the following: Determine the process index matrix to record the process identifiers at each position in the production plan of each machine; Determine the process time matrix to record the processing time of each process on each machine; Determine the process assignment vector to record the processes currently being executed by each machine; Determine the remaining time vector for each process, which is used to record the remaining time of the process currently being executed by each machine; Determine the process dependency matrix to record the dependencies between processes.

7. The method according to claim 1, characterized in that, The determination of (102) the set of currently executable processes to be processed based on the current state information and the DAG includes: Based on the current status information and the DAG, determine the processes that have not yet been completed in the process subset of the multiple jobs; For each incomplete process, query the preceding dependent processes in the DAG. If all preceding dependent processes of the process have been marked as completed according to the current status information, add the process to the set of currently executable pending processes.

8. The method according to claim 1, characterized in that, Also includes: Identify processes of the same type within the process subsets of different jobs; In the DAG, processes of the same type are merged into one node; When the merge node is executed, the same process in the different jobs is processed simultaneously by the same machine.

9. The method according to claim 1, characterized in that, It also includes the training process of the reinforcement learning model; during the training process, a pre-trained reinforcement learning model is obtained, and the pre-trained model is fine-tuned according to the number of machines and processes in the target production scenario.

10. The method according to claim 1, characterized in that, It also includes at least one of the following: When a machine malfunction is detected, the processing time of the processes being executed and those yet to be executed on the malfunctioning machine is set to zero. When an urgent order is detected, the new job is added to the cache queue to wait for loading; When a job cancellation is detected, all incomplete processes in the process subset of that job are marked as completed in the DAG.

11. The method according to claim 1, characterized in that, The reinforcement learning model makes decisions for each machine according to a preset time slot period. In each time slot, the machine that needs to make a decision at the moment decides on the process to be executed from the set of currently executable processes to be executed. The decision space of the reinforcement learning model includes idle actions, which indicate that the machine that needs to make a decision should not perform a procedure in the current time slot.

12. The method according to claim 11, characterized in that, The decision-making process for each machine also includes: when the machine that currently needs to make a decision is selected to be idle, the machine does not perform the process in the current time slot and participates in the decision-making process again in the next time slot.

13. The method according to claim 1, characterized in that, The workflow has a preset upper limit on the number of processes; When a new job arrives and the number of processes in the workflow is less than the maximum number of processes, the new job is added to the workflow; as well as When a new job arrives and the number of processes in the workflow is not less than the upper limit of the number of processes, the new job is added to the cache queue.

14. The method according to claim 1, characterized in that, Also includes: When a new job is added to the workflow, identify the processes in the process subset of the new job that are of the same type as the processes in the process subset of existing jobs; In the DAG, the processes in the process subset of the new job and the processes of the same type in the process subset of the existing job are merged into a single node, and the original dependencies of the processes in the process subset of the new job and the processes of the same type in the process subset of the existing job are transferred to the merged node.

15. The method according to claim 1, characterized in that, The reinforcement learning model is trained using the policy gradient method; the DAG is described in at least one of the following ways: Visualization graphs; adjacency lists; JSON / YAML format; or Cypher query language.

16. An electronic device comprising a processor (801) and a memory (802), said memory (802) storing a computer program, characterized in that, When the computer program is executed by the processor (801), it implements the method as described in any one of claims 1 to 15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 15.

18. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 15.

Images