Processing and assembling hybrid flexible job shop scheduling method based on reinforcement learning
By constructing a flexible work workshop scheduling model based on reinforcement learning, the scheduling problem in complex process environments is solved, efficient production scheduling is achieved, and scheduling quality is improved.
Patent Information
- Application Number
- CN202411664016.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-02-25
AI Technical Summary
The existing operation workshop scheduling methods are difficult to meet the personalized needs in complex process environments, resulting in problems such as backlog of product in-process inventory, reduced finished product output, and delivery delays.
The processing and assembly hybrid flexible operation workshop scheduling method based on reinforcement learning is adopted, and the production scheduling solution is generated by generating training sets, building reinforcement learning models, dual encoder and decoder parameterization, and using multi-PPO deep reinforcement learning framework.
Dynamic flexible work workshop scheduling in complex process environments is realized, the number of agents is reduced, model stability is enhanced, model complexity is reduced, and scheduling quality is improved.
Smart Images
Figure CN119511999A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of job shop scheduling, and in particular to a hybrid flexible job shop scheduling method for processing and assembly based on reinforcement learning. Background Art
[0002] With the increasingly fierce competition in the international market and the surge in customer customized products, the production model is evolving towards multiple varieties and small batches, and is gradually becoming the mainstream. Under this production model, the product is usually composed of multiple modular parts, and the processing of each part is completed through the processing process, and then the parts are assembled into finished products through the assembly process, so as to meet the customer's personalized customization needs.
[0003] However, in actual production, the arrival time of orders is uncertain, there are many types of parts, the process flow is complex, and the processing time also varies. It is difficult to control the production progress of each part in the traditional way. During the assembly process, a part is often not completed, which leads to product assembly obstruction, resulting in a series of problems such as backlog of work-in-process inventory, reduced finished product output, and delayed delivery. Therefore, it is necessary to realize the coordinated scheduling of processing and assembly processes in the dynamic scheduling process, so as to adapt to the current production reality of personalized needs and meet the scheduling of complex process environments. Summary of the invention
[0004] The present application provides a hybrid flexible job shop scheduling method for processing and assembly based on reinforcement learning to solve the problem that the existing job shop scheduling method cannot meet the needs of complex process environments.
[0005] The present application provides a hybrid flexible job shop scheduling method for processing and assembly based on reinforcement learning, the method comprising:
[0006] Generate a training set based on a mixed scheduling example of a flexible job shop and a flexible assembly shop; the training set includes the preceding process information, optional machine information, and processing / assembly time information of the process;
[0007] Constructing a reinforcement learning model based on Markov decision making and the training set; the reinforcement learning model includes an intelligent agent, an environment state representation module, an action selection module, a state transfer module, a benefit module and a strategy module;
[0008] Parameterizing the process selection action strategy and the machine selection action strategy in the reinforcement learning model through dual encoders and decoders;
[0009] The multi-PPO deep reinforcement learning framework based on proximal policy optimization is used to learn the process selection action strategy and the machine selection action strategy to generate a production scheduling plan.
[0010] This application combines machining process selection and assembly process selection into process action selection agents through the adjacency matrix, and also combines machining process machine allocation and assembly station selection into machine action selection agents, and then realizes flexible job shop production scheduling including processing and assembly through two agents. This method can reduce the number of agents, enhance the stability of the model, reduce the complexity of the model, and meet the dynamic flexible job shop scheduling under the conditions of multi-stage processing and assembly mixed process.
[0011] Optionally, the agent includes a process action selection agent and a machine action selection agent;
[0012] The process action selection agent is used to learn the process action selection strategy to perform process action selection;
[0013] The machine action selection agent is used to learn a machine action selection strategy to perform machine action selection.
[0014] The process action selection agent and the machine action selection agent work together to continuously optimize their decision-making process, thereby more flexibly adapting to complex process environments and improving scheduling quality.
[0015] Optionally, the environment state representation module is used to describe the global state at each time step; the global state includes the process local state and the machine local state;
[0016] The process local state is represented by a disjunctive graph; the machine local state is composed of the states of all machines at each time step.
[0017] Optionally, the action selection module is used to:
[0018] At the time step, a workpiece process action is selected in a set of processable processes according to the process selection action strategy to obtain a workpiece process action; wherein the preceding processes of all workpiece processes in the set of processable processes have been processed;
[0019] According to the machine selection action strategy, a machine action selection is performed in the set of processable machines of the workpiece process action to obtain a machine action.
[0020] Optionally, the state transfer module is used to:
[0021] At the time step, using the agent to perform the process selection action and the machine selection action;
[0022] adding a disjunctive arc according to the process selection action and the machine selection action to update the disjunctive graph as a new process local state;
[0023] Update the machine completion time and the processing time of the selected workpiece operation to obtain the new global status.
[0024] Optionally, the profit module is used to calculate the short-term profit of each time step according to the profit function to evaluate the scheduling quality.
[0025] Optionally, the strategy module is provided with a workpiece selection strategy and a machine selection strategy;
[0026] The workpiece selection strategy is used to select a process action according to a probability distribution generated in the workpiece process action space;
[0027] The machine selection strategy is used to select machine actions based on the probability distribution generated by the process action and the machine action space.
[0028] Optionally, the dual encoder uses a graph isomorphic network and a fully connected network to encode the local state of the process and the local state of the machine respectively; the decoder is used to perform decoding operations on each time step to construct a production scheduling plan.
[0029] Optionally, the Multi-PPO algorithm includes an evaluation function and two execution functions;
[0030] The evaluation function is used to guide the two execution functions to update;
[0031] The two execution functions learn the process selection action strategy and the machine selection action strategy respectively.
[0032] It can be seen from the above technical scheme that the present application provides a scheduling method for a hybrid flexible job shop for processing and assembly based on reinforcement learning, the method comprising: generating a training set according to a hybrid scheduling instance of a flexible job shop and a flexible assembly shop; the training set comprises the preceding process information of the process, optional machine information and processing / assembly time information of the process; constructing a reinforcement learning model based on Markov decision and the training set; the reinforcement learning model comprises an intelligent agent, an environment state representation module, an action selection module, a state transition module, a benefit module and a strategy module; parameterizing the process action selection strategy and the machine action selection strategy in the reinforcement learning model through a dual encoder and a decoder; adopting a multi-PPO deep reinforcement learning framework based on proximal policy optimization to learn the process selection action strategy and the machine selection action strategy to generate a production scheduling plan, so as to solve the problem that the existing job shop scheduling method cannot meet the complex process environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0034] Figure 1 It is a flow chart of the method for scheduling a hybrid flexible job shop for machining and assembly based on reinforcement learning as described in an embodiment of the present application;
[0035] Figure 2 It is a schematic diagram of the action selection process of the hybrid flexible job shop scheduling method for machining and assembly based on reinforcement learning described in an embodiment of the present application;
[0036] Figure 3 This is a process diagram for updating the process extraction graph of the hybrid flexible job shop scheduling method for machining and assembly based on reinforcement learning described in an embodiment of the present application. DETAILED DESCRIPTION
[0037] The embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following embodiments do not represent all implementation methods consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application.
[0038] In order to solve the problem that the existing job shop scheduling method cannot meet the complex process environment, the embodiment of the present application provides a hybrid flexible job shop scheduling method for processing and assembly based on reinforcement learning, see Figure 1 , the method comprising:
[0039] S100: Generate a training set according to a mixed scheduling instance of a flexible job shop and a flexible assembly shop.
[0040] The training set includes the process information of the preceding process, the optional machine information and the processing / assembly time information. Specifically, the training set may include d randomly generated examples, 50% of which are FJSP examples and 50% are FAJSP examples.
[0041] In the FJSP example, except for the first process which has no preceding process, the other processes have only one preceding process, as shown in Table 1.
[0042] Table 1 Information of a single workpiece in the FJSP example with a scale of 5×5 randomly generated
[0043]
[0044] In the FAJSP example, each process of the workpiece has 0 or more preceding processes. Except for the last process which has no subsequent process, the other processes have only one subsequent process, as shown in Table 2.
[0045] Table 2 Information of a single workpiece in the FAJSP example with a scale of 5×5 randomly generated
[0046]
[0047] S200: constructing a reinforcement learning model based on Markov decision making and the training set; the reinforcement learning model includes an intelligent agent, an environment state representation module, an action selection module, a state transfer module, a benefit module and a strategy module.
[0048] It should be understood that the Markov decision process-based reinforcement learning model consists of a tuple of the following five core elements: To define:
[0049] The state space represents the set of all possible environmental states. t Describe the working status of each machine in the workshop and the processing progress of the workpiece at time step t.
[0050] The action space contains the set of actions that all agents can perform. The agent takes actions at time step t according to the current state s. t Select an action a from the action space t .
[0051] P: state transition probability, defined from the current state s t Execute action a t Then transfer to the new state s t+1 probability.
[0052] r: Reward function, at each time step t, when action a is executed t and transfer to the new state s t+1 When , an immediate reward r is obtained from the environment t .
[0053] γ: Discount factor used to calculate the weight of long-term gains, indicating the importance of future rewards relative to current rewards.
[0054] In a multi-agent scenario, the strategy π(a|s) of each agent represents the probability of choosing action a in a given state s. The goal of reinforcement learning is to learn the strategy π to maximize the expected total benefit.
[0055] S300: Parameterizing the process action selection strategy and the machine action selection strategy in the reinforcement learning model through a dual encoder and a decoder.
[0056] Specifically, at time step t, two independent encoders work separately: one encoder is responsible for processing the input information related to the process and generating the local state encoding vector of the process; the other encoder is responsible for processing the input information related to the machine and generating the local state encoding vector of the machine. The decoder converts the encoded information into the probability distribution of process actions or machine actions based on the attention mechanism and the Softmax activation function, and selects process actions and machine actions from the corresponding probability distribution according to the decoding strategy. During the decoding process, in order to ensure the consistency of scheduling and meet the constraints of the previous process, the model adopts a mask strategy to avoid dispatching processes that have been scheduled or violate the constraints of the previous process and machines that cannot process the process.
[0057] S400: Using a multi-PPO deep reinforcement learning algorithm based on proximal policy optimization to learn the process selection action strategy and the machine selection action strategy to generate a production scheduling plan.
[0058] It should be understood that the multi-PPO deep reinforcement learning algorithm based on proximal policy optimization is used to learn the process selection action strategy and machine selection action strategy, which is to deal with the multi-action reinforcement learning problem by training the policy network. The trained deep reinforcement learning model is applied to the actual scheduling problem of the mixed processing and assembly workshop to provide a production scheduling solution.
[0059] This application combines machining process sorting and assembly process sorting into process sorting agents through the adjacency matrix, and also combines machining process machine allocation and assembly station selection into machine action selection agents, and then uses two agents to realize workshop production scheduling including machining and assembly. This method can reduce the number of agents, enhance the stability of the model, reduce the complexity of the model, and meet the dynamic flexible job shop scheduling under multi-stage machining and assembly mixed process conditions.
[0060] In some embodiments, the agent includes a process action selection agent and a machine action selection agent;
[0061] The process action selection agent is used to learn the process action selection strategy to perform process action selection;
[0062] The machine action selection agent is used to learn a machine action selection strategy to perform machine action selection.
[0063] The process action selection agent and the machine action selection agent work together to continuously optimize their decision-making process, thereby more flexibly adapting to complex process environments and improving scheduling quality.
[0064] In some embodiments, the state module is used to describe the global state at each time step; the global state includes the process local state and the machine local state;
[0065] The process local state is represented by a disjunctive graph; the machine local state is composed of the states of all machines at each time step.
[0066] It should be understood that the process local state From the disjunctive graph express. represents all the disjunctive arcs that have been assigned directions at time step t, Represents the remaining disjunctive arcs. Each node in the disjunctive graph The original features are composed of three one-dimensional features [LB t (O j,i ),ES t (O j,i ),I t (O j,i )] Common description: Among them LB t (O j,i ) indicates that process O has been scheduled j,i The completion time of the unscheduled process or the minimum estimated completion time of the unscheduled process LB t (O j,i-1 )+min(p j,i,k ,k∈A j,i ). t (O j,i ) indicates that process O has been scheduled j,i The actual start time of the work or the earliest possible start time of the unscheduled process I t (O j,i ) means O j,i Whether it is scheduled. If it is scheduled, it is 1, otherwise it is 0.
[0067] The local state of the machine It consists of the states of all machines at time step t. For each machine M k , whose state is described by a two-dimensional feature vector, expressed as [T t (M k ),p j,i,k ], where T t (M k ) indicates that at time step t, machine M k The time to complete the last process, P j,i,k Indicates the selected process O j,i On machine M k The time required for processing is k If the process cannot be processed, the process set Aj,i The average processing time of all machines in this process replace.
[0068] The global state can fully reflect the operation status of the system at any time. The local state of the process is represented by a disjunctive graph, which graphically displays the dependency relationship between processes.
[0069] In some embodiments, the action selection module is used to:
[0070] At the time step t, a workpiece process action is selected in the process set according to the process selection action strategy to obtain a workpiece process action; wherein the preceding processes of all the workpiece processes in the process set have been processed;
[0071] According to the machine selection action strategy, a machine action selection is performed in the set of processable machines of the workpiece process action to obtain a machine action.
[0072] It should be understood that at time step t, the action is selected by the workpiece process action a o ∈A o and the machine chooses action a m ∈A m Composition, A o A represents the set of processes that can be selected for processing at time step t, and the predecessor processes of all processes in this set have been processed. m Indicates the selected process action a o The process selection is first performed, and then the machine selection is performed based on the process selection action. The selection process is as follows: Figure 2 shown.
[0073] In some embodiments, the state transfer module is used to:
[0074] At the time step t, using the agent to perform the process action selection and the machine action selection;
[0075] adding a disjunctive arc according to the process action selection and the machine action selection to update the disjunctive graph as a new process local state;
[0076] Update the machine completion time and the processing time of the selected workpiece operation to obtain the new global status.
[0077] It should be understood that at time step t, the agent performs action a t =(a o ,a m ), and according to a o and a m Add disjunctive arcs and update the disjunctive graph as the new local state of the process The update process is as follows Figure 3 As shown. At the same time, update the machine completion time T t+1 (M k ) and the selected process a o Processing time p j,i,k , get the new machine local state
[0078] In some embodiments, the profit module is used to calculate the short-term profit of each time step according to the profit function to evaluate the scheduling quality.
[0079] It should be understood that the calculation formula of the profit function is:
[0080] γ t =-(C(s t+1 )-C(s t ));
[0081] Among them, C(s t )=max(LB t (O j,i )) represents the maximum completion time at time step t.
[0082] In some embodiments, the strategy module has a workpiece selection strategy and a machine selection strategy inside;
[0083] The workpiece selection strategy is used to select a process action according to a probability distribution generated in the workpiece process action space;
[0084] The machine selection strategy is used to select machine actions based on the probability distribution generated by the process action and the machine action space.
[0085] It should be understood that the policy function π θ (a o ,a m |s) Strategy selection by workpiece process and machine selection strategy Composition, where θ o Select a set of strategic parameters for the workpiece process, θ m Select a set of policy parameters for the machine, θ = [θ o ,θ m ] represents the parameter set of the entire strategy. Strategy π θ (a o ,a m |s) First, according to the workpiece process action space A o The resulting probability distribution selects process action a o , then based on process action a o and the machine action space A m The resulting probability distribution selects the machine action am .
[0086] In some embodiments, the dual encoder uses a graph isomorphic network and a fully connected network to encode the local state of the process and the local state of the machine respectively; the decoder is used to perform a decoding operation on each time step to construct a production scheduling plan.
[0087] It should be understood that the dual encoder uses a graph isomorphic network and a fully connected network to respectively encode the local state s of the process o and the machine local state s m At time step t, the local state of the process From the disjunctive graph Definition: C is a set of connecting arcs, representing the constraints of the preceding processes from the same workpiece; D is a set of disjunctive arcs, each arc connecting a pair of processes that can be processed by the same machine. Each node is encoded and embedded through L layers of GIN. Each layer of GIN is represented as:
[0088]
[0089] in, is the embedding of node v in the l-th layer GIN; is the original input feature; is the lth layer of multilayer perceptron; θ l is the parameter of this layer; ε (l) is a learnable parameter; N(v) represents the neighborhood set of node v.
[0090] Use fully connected layers to encode the local state of the machine s m At time step t, the local state of each machine node is is expressed as [T t (M k ),p j,i,k ], the node embedding is represented as The pooled vector is denoted as u t .
[0091] The decoder performs decoding operations on each time step t, gradually building a complete production scheduling solution. Wherein, time step t∈{1,...,|O}, |O| represents the total number of processes in the example. At time step t, the process decoder and machine decoder operate independently, based on the current state and Select Action and and Together they form the complete action at time step t:
[0092]
[0093] The process decoder and the machine decoder have the same structure but do not share parameters.
[0094] In the decoding stage, the workpiece process action coefficients are output through the MLP layer and machine motion coefficient The calculation is as follows:
[0095]
[0096] Among them, ·||· represents the connection operation. and Set to -∞ to mask out processes that have been scheduled or violate the constraints of their predecessor processes, as well as machines that are incompatible with the selected process. and Normalized, the calculation is as follows:
[0097]
[0098] Then based on and Apply sampling strategy or greedy strategy prediction and
[0099] In some embodiments, the Multi-PPO algorithm includes an evaluation function and two execution functions;
[0100] The evaluation function is used to guide the two execution functions to update;
[0101] The two execution functions learn the process selection action strategy and the machine selection action strategy respectively, which are used for process action selection and machine action selection. The selection process is: process action selection strategy π θ (a o |s) Select a process action based on the state, and the machine action selection strategy π θ (a m |s,a o ) Select a machine based on the selected operation.
[0102] It can be seen from the above technical scheme that the embodiment of the present application provides a hybrid flexible job shop scheduling method for processing and assembly based on reinforcement learning, the method comprising: generating a training set according to a hybrid scheduling instance of a flexible job shop and a flexible assembly shop; the training set includes the preceding process information of the process, optional machine information and processing / assembly time information; constructing a reinforcement learning model based on Markov decision and the training set; the reinforcement learning model includes an intelligent agent, an environment state representation module, an action selection module, a state transition module, a benefit module and a strategy module; configuring the process selection action strategy and the machine selection action strategy in the reinforcement learning model through a dual encoder and a decoder; using a multi-PPO deep reinforcement learning algorithm based on proximal strategy optimization to learn the process selection action strategy and the machine selection action strategy to generate a production scheduling plan to solve the problem that the existing job shop scheduling method cannot meet the complex process environment.
[0103] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.
Claims
1. A hybrid flexible job shop scheduling method for processing and assembly based on reinforcement learning, characterized in that: The method comprises: Generate a training set based on a mixed scheduling example of a flexible job shop and a flexible assembly shop; the training set includes the preceding process information, optional machine information, and processing / assembly time information of the process; Constructing a reinforcement learning model based on Markov decision making and the training set; the reinforcement learning model includes an intelligent agent, an environment state representation module, an action selection module, a state transfer module, a benefit module and a strategy module; Parameterizing the process selection action strategy and the machine selection action strategy in the reinforcement learning model through dual encoders and decoders; The multi-PPO deep reinforcement learning framework based on proximal policy optimization is used to learn the process selection action strategy and the machine selection action strategy to generate a production scheduling plan.
2. The method for scheduling a hybrid flexible job shop for machining and assembly based on reinforcement learning according to claim 1 is characterized in that: The intelligent agent includes a process action selection intelligent agent and a machine action selection intelligent agent; The process action selection agent is used to learn the process action selection strategy to perform process action selection; The machine action selection agent is used to learn a machine action selection strategy to perform machine action selection.
3. The method for scheduling a hybrid flexible job shop for machining and assembly based on reinforcement learning according to claim 1 is characterized in that: The environment state representation module is used to describe the global state at each time step; the global state includes the process local state and the machine local state; The process local state is represented by a disjunctive graph; the machine local state is composed of the states of all machines at each time step.
4. The method for scheduling a hybrid flexible job shop for machining and assembly based on reinforcement learning according to claim 3 is characterized in that: The action selection module is used to: At the time step, a workpiece process is selected from a set of processable processes according to the process selection action strategy to obtain a process to be arranged at the current time step; wherein the preceding processes of all workpiece processes in the set of processable processes have been processed; Machine selection is performed in the set of processable machines of the process to be arranged according to the machine selection action strategy to obtain the processing machine of the process to be arranged.
5. The method for scheduling a hybrid flexible job shop for machining and assembly based on reinforcement learning according to claim 4 is characterized in that: The state transfer module is used for: At the time step, using the agent to perform the process action selection and the machine action selection; adding a disjunctive arc according to the process selection action and the machine selection action to update the disjunctive graph as a new process local state; Update the machine completion time and the processing time of the selected workpiece operation to obtain the new global status.
6. The method for scheduling hybrid flexible job shops for machining and assembly based on reinforcement learning according to claim 1 is characterized in that: The profit module is used to calculate the short-term profit of each time step according to the profit function to evaluate the scheduling quality.
7. The method for scheduling hybrid flexible job shops for machining and assembly based on reinforcement learning according to claim 1 is characterized in that: The strategy module is internally provided with a workpiece selection strategy and a machine selection strategy; The workpiece selection strategy is used to select a process action according to a probability distribution generated in the workpiece process action space; The machine selection strategy is used to select machine actions based on the probability distribution generated by the process actions and the machine action space.
8. The method for scheduling hybrid flexible job shops for machining and assembly based on reinforcement learning according to claim 1 is characterized in that: The dual encoder uses a graph isomorphic network and a fully connected network to encode the local state of the process and the local state of the machine respectively; the decoder is used to perform a decoding operation on each time step to construct a production scheduling plan.
9. The method for scheduling hybrid flexible job shops for machining and assembly based on reinforcement learning according to claim 1 is characterized in that: The Multi-PPO algorithm includes an evaluation function and two execution functions; The evaluation function is used to guide the two execution functions to update; The two execution functions learn the process selection action strategy and the machine selection action strategy respectively.