Real-time order receiving and scheduling method and system for flexible job shop

Abstract

The invention relates to the technical field of cloud manufacturing, in particular to a real-time order receiving and scheduling method and system for a flexible job shop. Comprising the following steps: extracting related data from a production site, and constructing a flexible job shop environment model according to the related data; an agent decision model is constructed, a corresponding DQN algorithm model is constructed, and the DQN algorithm model comprises an order acceptance DQN and a scheduling DQN; on the basis of interaction with the intelligent agent, training an order acceptance DQN algorithm model and a scheduling DQN algorithm model, the training of the order acceptance DQN being completed on the basis of the scheduling DQN having a good effect; and deploying the trained order acceptance DQN and the scheduling DQN to make a decision. According to the method, comprehensive fine consideration of the external production capacity and the expected available production capacity of the actual production workshop is added, a high-customization order-oriented order acceptance decision and production logistics collaborative scheduling real-time comprehensive scheduling model is established, the deviation of real-time acceptance decision is reduced, and more efficient and scientific management and control of the production process are realized.

Description

Technical Field

[0001] The present invention relates to the field of cloud manufacturing technology, and in particular to a real-time order acceptance and scheduling method and system for a flexible job shop. Background Art

[0002] In the competitive market environment, customer customization needs are increasing. At the same time, the continuous development of new generation high-tech technologies such as cloud computing, Internet of Things (IoT) and big data provides support for production to meet customer customization needs. The production model based on order customization has been widely used because it can help enterprises realize customized demand-driven production, meet customer personalized needs and gain advantages in competition. However, in the process of open advanced manufacturing mode to improve resource utilization flexibility by decomposing production tasks and combining production capabilities, considering the manufacturer's resource capacity constraints and the pursuit of production quality and efficiency, it may not be optimal for manufacturers to accept all orders. Therefore, real-time order acceptance and scheduling (r-OAS) is an important issue in the order-based production environment. Facing the dynamic and complex production environment, it is necessary to decide which orders to accept and update the scheduling plan in time to meet the requirements of order-oriented manufacturing and gain a competitive advantage.

[0003] With the development of intelligence and agility in the global production environment, the dynamics and uncertainty of order arrival in order-based production have become more prominent, which has also put forward higher requirements for real-time and flexible order acceptance and scheduling. While various open advanced manufacturing models based on order customization enhance the ability of production systems to cope with complex changes, the dynamics and complexity of order acceptance and scheduling in production systems have been further enhanced. This is mainly reflected in the uncertainty of order arrival, the personalization of work, and the higher frequency of order changes. At the same time, the introduction of shared manufacturing resources in the decision-making process of order acceptance in a more open manufacturing environment breaks through the limitations of inherent manufacturing resources and further expands the decision-making space for order acceptance. The current method of accepting real-time orders lacks comprehensive and detailed consideration of external production capacity and the expected available production capacity of the actual production workshop, which may cause certain deviations in real-time acceptance decisions. Therefore, for flexible manufacturing workshops in real-time dynamic production environments, establishing a real-time integrated scheduling model for order acceptance decisions and production logistics collaborative scheduling for highly customized orders will enable more efficient and scientific control of the production process, which is of great significance for actual production. Summary of the invention

[0004] The purpose of the present invention is to provide a new method for real-time order acceptance and scheduling in an open manufacturing environment, taking into account the comprehensive and detailed consideration of external production capacity and the expected available production capacity of the actual production workshop, and establishing a real-time comprehensive scheduling model for order acceptance decisions and production logistics collaborative scheduling for highly customized orders, thereby reducing the deviation of real-time acceptance decisions and achieving more efficient and scientific management and control of the production process.

[0005] The real-time order acceptance and scheduling method of the flexible job shop of the present invention comprises the following steps:

[0006] Step 1 extracts relevant data from the production site and builds a flexible job shop environment model based on the relevant data;

[0007] Step 2 constructs an agent decision model, wherein the multi-agent decision model includes an order acceptance agent and a production logistics collaborative scheduling agent, which is used to simulate the execution process of order acceptance and scheduling in the production system;

[0008] The order acceptance agent model includes state characteristics, actions, and reward functions;

[0009] The production logistics collaborative scheduling agent model includes state characteristics, actions, and reward functions;

[0010] Step 3: Based on the constructed order acceptance agent and production logistics collaborative scheduling agent, a corresponding DQN algorithm model is constructed, wherein the DQN algorithm model includes an order acceptance DQN and a scheduling DQN;

[0011] Step 4: Based on the interaction with the agent, the order acceptance DQN and scheduling DQN algorithm models are trained. The training of the order acceptance DQN is completed on the basis of the scheduling DQN which has a better effect.

[0012] Step 5 deploys the trained order acceptance DQN and scheduling DQN. The order acceptance DQN makes a decision on whether to accept the order, and the scheduling DQN makes a decision on the production scheduling of the order.

[0013] Furthermore, in step 1, the relevant data includes order information and status, machine information and status, workpiece information and status, process information and status, and AGV information and status.

[0014] Furthermore, in the job shop environment model in step 1, the randomly arriving orders are represented as O = [O1, O2, ... n …O N ], Order O n The characteristic attributes of include its delivery deadline ddl, arrival time ontime, price price and delay penalty coefficient pu;

[0015] Order O n All the tasks in the group form a task set Right now Indicates order O n The dth job in Contains a set of processes, and the process set is represented as

[0016] All processes can be divided into K categories, and the process type corresponds to the equipment type one by one, that is, the type attribute k∈[1,2,…K] of the process or equipment, and the order O n The number of each type of process in is expressed as Indicates order O n The number of processes with internal process type k;

[0017] The device set is represented by M = [M1, M2, ... M c …M C ],A=[A1,A2…A q …A Q ] represents the logistics task set, V = [V1, V2…V r …V R ] represents the AGV set;

[0018] The called open production resource has a time window constraint, which is defined by the start time winStart and the end time winEnd;

[0019] Order O n The acceptance decision is expressed as As shown below:

[0020]

[0021] Furthermore, the construction of the multi-agent decision-making model follows the following strategy:

[0022] Both the order acceptance decision and the real-time production logistics collaborative scheduling are modeled as Markov decision processes;

[0023] In the order acceptance agent under open manufacturing resources, the decision point occurs when a new order arrives. When a new order arrives, the order acceptance agent decides whether to accept the order based on the production status characteristics provided by the production system and the characteristics of the new order, and determines the allocation of open production resources. The plan minimizes production delays, including estimated costs and predicted capacity gaps.

[0024] The scheduling agent obtains production tasks based on the decision of the order acceptance agent and reschedules production and logistics tasks according to the current production status of the flexible job shop. The goal of the scheduling agent is to minimize the total delayed delivery time of all orders and set the corresponding reward function accordingly.

[0025] The scheduling agent assigns all pending processes to production equipment and determines the estimated production processing time. During the production scheduling process, if a production process has a subsequent process, a material handling task is generated and assigned to the earliest available AGV to complete the logistics scheduling within the production flexible operation workshop;

[0026] The scheduling agent completes the formulation of the scheduling plan and updates the scheduling plan to the production system. The production system executes according to the plan and transmits the current status of the production system to the order acceptance agent and the scheduling agent in real time when necessary.

[0027] Furthermore, the state characteristics of the order acceptance agent model include:

[0028] The urgency of the order O , which represents the ratio of the processing time of all processes in an order to the remaining processing time;

[0029] Expected revenue, Rp, represents the price of all accepted orders minus expected costs;

[0030] The expected rate of return, Pr, represents the ratio of the expected total benefits to the expected total costs;

[0031] Standardized production execution cost nCost re , which is the ratio of the expected maximum return to the production execution cost;

[0032] Normalized maximum cost index nCost for calling open production resources po ;

[0033] Normalized maximum delay cost index nCost op , represents the ratio of expected maximum revenue to expected deferred cost under basic capacity;

[0034] The average production capacity gap of orders Dv O , represents the average difference between the production demand and the actual available production capacity for all types of work processes specified in the order;

[0035] The normalized return index nEarnp represents the ratio of the expected maximum return to the average revenue of the accepted orders;

[0036] The actions of the order acceptance agent model include:

[0037] Action 1: Do not accept orders and do not call open production resources;

[0038] Action 2: Accept the order and call on external manufacturing resources, with the goal of minimizing the estimated cost;

[0039] The state transitions of the order acceptance agent model include:

[0040] At the initial time t0, the production system is in the initial state s0, at which time the arrival status of all orders is not arrived; the subsequent state is s k Transfer to k+1 During the process, time is t k Transfer to k+1, the agent selects an action a for the kth order and decides whether to accept the order. The state value is s k Transfer to k+1 And generate a reward value r k+1 , until the state is transferred to the terminal state t=t0+T to obtain the acceptance decision of all orders in period T;

[0041] The order-taking agent model uses the normalized expected total revenue over the cycle time as a reward value.

[0042] Furthermore, the state characteristics of the scheduling agent model include:

[0043] Average machine utilization U ave ;

[0044] Standard deviation of machine utilization U std ;

[0045] Average order completion rate CO ave ;

[0046] The standard deviation of the completion rate of each order process CO std ;

[0047] The actual delay time of the order D act ;

[0048] In the scheduling agent model, the selection criteria for production equipment are defined by the following two rules:

[0049] Device matching rule M 1: Select the machine with the smallest earliest available time TC at time t;

[0050] Device matching rule M 2: Select the machine with the lowest utilization U at time t;

[0051] The actions of the scheduling agent model include:

[0052] Action 1: Prioritize the next unscheduled process of the workpiece with the largest delay time, otherwise, the process selection is determined according to the SOT rule. In the selection of equipment, the machine with the earliest available time is given priority, that is, rule M 1;

[0053] Action 2: Prioritize the next unscheduled process of the workpiece with the largest delay time. Otherwise, the process selection follows the earliest delivery time rule. The equipment selection rule is rule M 1;

[0054] Action 3: Prioritize the next unscheduled process of the workpiece with the largest delay time. Otherwise, the process selection follows the STR / OP rule, and the equipment selection rule follows rule M 1;

[0055] For actions 4, 5, and 6, the process selection rules refer to SOT, EDD, and STR / OP respectively, and the equipment selection rules are all rule M 2;

[0056] The state transition of the scheduling agent model includes:

[0057] At the initial moment, the scheduling system is in the initial state s0, at which time the scheduling status of all processes is unscheduled, and all machines are in the idle state. t Transfer to t+1 During the process, the agent selects an action a, matches a device with a process, and the state value s t Transfer to t+1 And generate a reward value r t+1 , until the state is transferred to the terminal state to obtain the scheduling plan for all processes.

[0058] The scheduling agent model uses a logarithmic function to minimize the delivery time delay as the basis for defining the reward function; at the same time, equipment utilization is used as an auxiliary indicator to consider providing positive rewards for scenarios where the delay time is extended but the equipment load is high.

[0059] Furthermore, the construction of the DQN algorithm model follows the following strategy:

[0060] The construction of the DQN model includes two parts: order acceptance DQN model construction and scheduling DQN model construction. In the order acceptance DQN, the model's goal is to determine the order acceptance combination that arrives in the cycle to obtain a better overall return; in the scheduling DQN, the decision-making goal is to match production equipment for each production task to minimize production delay time;

[0061] The specific construction steps of the two DQNs are as follows:

[0062] Construct the Q deep neural network structure of order acceptance DQN to perceive the state characteristics of the production system, including input layer, hidden layer, and output layer; the number of input layer nodes corresponds to the number of state characteristics of the order acceptance agent model, and the number of output layer nodes corresponds to the number of actions of the order acceptance agent model. Adopt the Adam optimization algorithm to establish the target neural network with the same structure, establish an experience replay pool D with a capacity of N, and store the experience (s t ,a t d,r t ,s t+1) is used for experience playback;

[0063] Construct a Q deep neural network structure for scheduling DQN to perceive the state characteristics of the scheduling system, including input layer, hidden layer, and output layer; the number of input layer nodes corresponds to the number of state characteristics of the scheduling agent model, and the number of output layer nodes corresponds to the number of actions of the scheduling agent model. The Adam optimization algorithm is used; the target neural network is established with the same structure, and an experience replay pool D with a capacity of N is established to store experience (s t ,a t d,r t ,s t+1 ) is used for experience playback;

[0064] In the production system, when a new order arrives, the order acceptance DQN perceives the production status characteristics of the production system and makes an order acceptance decision. The scheduling DQN perceives the current scheduling system status and reschedules the processes of the orders for which the acceptance decision is made and the processes that are not produced in the current workshop. The state transition process of the production system is jointly affected by the decisions made by the order acceptance DQN and the scheduling DQN.

[0065] Furthermore, in step 4, the steps for training the DQN algorithm model are as follows:

[0066] In the training process of the DQN model, the scheduling DQN is first trained to ensure that the DQN can produce a scheduling solution with a better delay time;

[0067] The scheduling training set is input into the scheduling DQN model for training. Each training is conducted on one case in the training set, and each training traverses all the production tasks in the case for scheduling. As the training progresses, the Q network is updated based on the target value, and the Q neural network is updated to the target neural network every n steps. The training gradually converges, the degree of exploration slowly decreases, and the probability of selecting the exploration action gradually decreases as the greed coefficient e decreases.

[0068] Based on the trained production logistics collaborative scheduling DQN, the order acceptance decision DQN is trained. It estimates the production cost to make further order acceptance decisions.

[0069] The order acceptance training set is input into the order acceptance DQN model for training. Each training session trains one case in the training set. Each training session traverses all orders in the case to make acceptance decisions. The state value of the order acceptance decision is provided by the production environment. The state value of the production environment changes with the progress of time steps according to the latest scheduling plan of the scheduling DQN. As the training progresses, the Q network is updated based on the target value. The Q neural network is updated to the target neural network every n steps. The training gradually converges, the degree of exploration slowly decreases, and the probability of selecting the exploration action gradually decreases with the decrease of the greed coefficient e.

[0070] The present invention also discloses a real-time order acceptance and scheduling system for a flexible job shop, including a once-connected input module and a decision module, wherein the input module is used to input characteristic attribute data of an order to be decided, and the decision module makes an implementation decision on the acceptance of the order and subsequent production scheduling according to the aforementioned method.

[0071] The beneficial effects of the present invention are:

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] The present invention provides an order acceptance decision method that takes into account available external resources and production demand gaps. It comprehensively considers the production capacity provided by the real-time status of the workshop and the production capacity demand of the newly arrived orders, combines the production capacity provided by external resources, and takes the actual production capacity of the workshop as the basis to reasonably allocate the production capacity of the enterprise and formulate a scheduling plan, weigh the benefits and costs of the orders, and selectively accept the orders, which plays an important role in improving long-term profits and competitive advantages.

[0074] Based on the problem characteristics of the r-OAS problem, the present invention proposes a multi-agent method, constructs a Markov decision process model of each agent, and provides a real-time solution for order acceptance and scheduling through the collaboration of the order acceptance agent and the scheduling agent, further realizing the continuous steady-state and efficient operation of the production system under highly dynamic task requirements.

[0075] The present invention solves the r-OAS problem in an open manufacturing environment. Compared with the limitations of traditional heuristic algorithms in terms of timely response speed in complex dynamic scenarios, the present invention shows a higher level of responsiveness when making r-OAS decisions and can effectively adapt to the high dynamics of an open manufacturing environment. It has autonomous learning and self-evolution capabilities and strong generalization, and provides a solution method for r-OAS under open production capabilities, effectively improving the solution efficiency.

[0076] In particular, the present invention proposes a technical solution to the order acceptance decision problem by integrating the open production capacity provided by external open production resource leasing, that is, considering the coordination of internal and external manufacturing resources, improving resource utilization and effectively enhancing profit potential.

[0077] The present invention provides a multi-agent model based on deep Q network (DQN) to solve the real-time order acceptance and scheduling problem. The multi-agent DQN model is trained in the interaction process between the order acceptance agent and the scheduling agent, and the order acceptance agent and the scheduling agent in the multi-agent are respectively optimized by establishing a Markov model. It can jointly optimize the order acceptance decision and the scheduling scheme at the time of new order arrival, and provide a faster and more sophisticated solution method for the order acceptance decision and production logistics collaborative scheduling problem of the flexible workshop with real-time order arrival. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flow chart of the real-time order acceptance and scheduling method of the flexible job shop in the embodiment of the present invention;

[0079] Figure 2 This is a comparison chart of the execution results of the scheduling agent in Experimental Example 1 of the embodiment of the present invention;

[0080] Figure 3 This is a comparison chart of the execution results in Experimental Example 2 of the embodiment of the present invention. DETAILED DESCRIPTION

[0081] When solving the real-time order acceptance and scheduling problem in a highly dynamic open advanced manufacturing environment, the following assumptions are made in this example:

[0082] (1) Delayed delivery of orders will incur delay costs and inventory costs, while leasing external production equipment will incur rental costs.

[0083] (2) At any given time, a piece of production equipment can handle at most one process, and an automatic guided vehicle (AGV) can complete at most one transportation task at the same time.

[0084] (3) Orders cannot be split. An order can contain multiple jobs, and the decision to accept a production task is made on a per-order basis.

[0085] (4) The logistics time of any AGV between two points remains unchanged. In the process of product flow collaborative scheduling, the moving distance between two working locations of the AGV is simplified to a fixed distance.

[0086] (5) Each piece of production equipment has a specific type of process to which it belongs. The type of production equipment corresponds one-to-one to the type of process.

[0087] In addition, in this example, the real-time order acceptance and decision-making and scheduling problem of a flexible job shop is defined as follows:

[0088] In the initial environment, the enterprise has a set of accepted orders and a set of randomly arriving orders. The production requirements of the orders are first met by the remaining available production resources of the enterprise. When the production demand exceeds the production capacity of the enterprise, the manufacturer can use external open production resources within the profit range to meet the production requirements of these orders.

[0089] The task of the order acceptance agent is to make a decision on whether to accept each order that arrives and determine the scheduling plan of flexible manufacturing resources to maximize the overall profit. Among them, the production equipment corresponds to the type of production process one by one and can handle any process of its corresponding type. Specifically, the called open production resources have time window constraints, which are defined by their available start time and end time. Each process has a type, and its start time is constrained by the job chain. The workpiece is associated with the order it belongs to and the set of processes included. Based on the order acceptance decision and the rental plan of the external open production resources, the scheduling agent determines the allocation relationship between production tasks and production equipment in the production process, and the allocation relationship between logistics tasks and transportation equipment, with the goal of minimizing delay time.

[0090] like Figure 1 As shown, this embodiment discloses a method for real-time order acceptance and scheduling of a flexible job shop based on deep reinforcement learning, which is performed in the following steps:

[0091] Step 1: Extract relevant data from the production site, including order information and status, machine information and status, workpiece information and status, process information and status, and AGV information and status, and build a flexible job shop environment model based on the relevant data.

[0092] The real-time order acceptance, decision-making and scheduling production model of the flexible job shop is as follows:

[0093] The enterprise has a set of randomly arriving orders, represented by O = [O1, O2, ... n …O N ], Order O n The characteristic attributes of order O include its delivery deadline ddl, arrival time ontime, price price and delay penalty coefficient pu; n All the tasks in the group form a task set Right now Indicates order Q n The dth job in Contains a set of processes, and the process set is represented as All processes can be divided into K categories, and the process type corresponds to the equipment type one by one, that is, the type attribute k∈[1,2,…K] of the process or equipment, and the order O n The number of each type of process in is expressed as Indicates order O n The number of processes with internal process type k, the equipment set of the enterprise is expressed as M = [M1, M2, ... M c …M C ], in the process of production logistics scheduling, A=[A1,A2…A q …A Q] represents the logistics task set, V = [V1, V2…V r …V R ] represents an AGV collection; the open production resource called has a time window constraint, defined by the start time winStaty and the end time winEnd. Order O n The acceptance decision is expressed as As shown below:

[0094]

[0095] In the real-time order acceptance and scheduling problem, the goal of the order acceptance decision model is to maximize the overall profit margin f OA :

[0096]

[0097] The total production cost expressed as Cost mainly comes from three aspects: the production cost C required for the manufacturing process re , the cost of using open production resources C po and penalty for late delivery C op , as shown below:

[0098] Cost = C re +C po +C op (3)

[0099] The goal of the production logistics collaborative scheduling model is to minimize the total delay time f OS :

[0100]

[0101] Among them, order O n Actual delivery time It is expressed as the latest completion time of the workpiece in the order.

[0102] Step 2: Construct a multi-agent decision-making model, including the order acceptance agent O-Agent and the scheduling agent S-Agent.

[0103] When a new order O arrives, O-Agent decides whether to accept the order based on the production status characteristics provided by the production system and the characteristics of the new order. S-Agent reschedules production and logistics tasks based on the decision of the order acceptance agent O-Agent.

[0104] In this example, both the order acceptance decision and the real-time production logistics collaborative scheduling under open production resources are modeled as Markov decision processes.

[0105] In the order acceptance agent under open manufacturing resources, the decision point occurs when a new order arrives. When a new order arrives, the order acceptance agent decides whether to accept the order based on the production status characteristics provided by the production system and the characteristics of the new order, and determines the allocation of open production resources. The plan minimizes production delays, including estimated costs and predicted capacity gaps.

[0106] The scheduling agent obtains production tasks based on the decision of the order acceptance agent and reschedules production and logistics tasks according to the current production status of the flexible job shop. The goal of the scheduling agent is to minimize the total delayed delivery time of all orders, and the model sets the corresponding reward function based on this. The scheduling agent assigns all pending processes to production equipment and determines the estimated production processing time. During the production scheduling process, if a production process has a subsequent process, a material handling task is generated and assigned to the earliest available AGV to complete the logistics scheduling within the production flexible job shop.

[0107] The scheduling agent completes the formulation of the scheduling plan and updates the scheduling plan to the production system. The production system executes according to the plan and transmits the current status of the production system to the order acceptance agent and the scheduling agent in real time when necessary.

[0108] Specifically, the order acceptance agent model includes state characteristics, actions, and reward functions. The specific construction strategy is as follows:

[0109] Step 2.1: Construct the state features of O-Agent.

[0110] The parameters used are as follows:

[0111] PT k : The processing time of the process type k

[0112] Equipment M c The maximum completion time of scheduled orders on

[0113] t: current time

[0114] l k : Unit cost of the process type k

[0115] At time t, for order O n Equipment M c The remaining capacity is calculated by the following formula.

[0116]

[0117] u M : Equipment utilization, that is, the working time D of equipment M workTotal working time D On The ratio is calculated by the following formula.

[0118]

[0119] Status feature 1:

[0120] Order O n Urgency:

[0121]

[0122] Status feature 2:

[0123] Expected Total Revenue:

[0124]

[0125] Status feature 3:

[0126] Estimated rate of return:

[0127]

[0128] Status feature 4:

[0129] Standardized Order O n Production execution costs:

[0130]

[0131] Among them, order O n Expected production costs:

[0132]

[0133] Order O n Expected maximum return:

[0134]

[0135] At the current time t, order O n The production capacity requirement for process type k is expressed as

[0136]

[0137] 0-1 variables Indicates device M c Is the type of k:

[0138]

[0139] Status feature 5:

[0140] Order O nStandardized open resource access cost index:

[0141]

[0142] in, Indicates order O n For the capacity gap of process type k, pc k Represents the unit call cost of a device of type k.

[0143] Status feature 6:

[0144] Order O n The normalized maximum delay cost index is:

[0145]

[0146]

[0147] Status feature 7:

[0148] Order O n Average production capacity gap:

[0149]

[0150] At time t, order O n The capacity gap corresponding to equipment type k is expressed as

[0151]

[0152] Status Feature 8:

[0153] Order O n Standardized return index:

[0154]

[0155] Where aveEarn represents the average revenue of all orders received up to time t:

[0156]

[0157] Step 3.2 Constructing O-Agent actions

[0158] The order acceptance decision is expressed as As shown below:

[0159]

[0160] Consider the actions in the order-taking problem for opening a production facility lease:

[0161] Action 1: The rental quantity set rM←{0} of different types of production equipment.

[0162] Action 2: Call on external manufacturing resources at the lowest estimated cost.

[0163] Step 3.3 Construct the reward function of O-Agent

[0164] At the initial time t0, the production system is in the initial state s0. The subsequent state is determined by s k Transfer to k+1 During the process, time is t k Transfer to k+1 , the agent selects an action a for the kth order, and the state value s k Transfer to k+1 And generate a reward value r k+1 The goal of the order acceptance problem is to make the total profit f OA To maximize, the reward function is set up as follows:

[0165] [1] Determine whether time t reaches period T;

[0166] [2] If the cycle T is reached, the reward value aR t Returns the ratio of expected return to all return values ​​w e ;

[0167] [3] If the period T is not reached, the reward value aR t Returns 0.

[0168] Step 4. The steps to build S-Agent are as follows:

[0169] Step 4.1 Construct the state characteristics of S-Agent

[0170] Status feature 1: Average machine utilization

[0171]

[0172] Status feature 2: Standard deviation of machine utilization

[0173]

[0174] Status feature 3: Average order completion rate

[0175]

[0176] Order O n The proportion of completed work in the o

[0177] Status feature 4: Standard deviation of the completion rate of each order process

[0178]

[0179] Status feature 5: Actual delay time of the order

[0180]

[0181] Order O n The actual delivery time is expressed as

[0182] Step 4.2 Constructing the actions of S-Agent

[0183] In the production logistics coordination scheduling model, the two main decision-making parts are as follows:

[0184] 1) Assign the production process to be processed to the machine, decision variables Defined in formula (28);

[0185] 2) Assign material handling tasks to AGVs, decision variables Defined in formula (29).

[0186]

[0187] The logistics scheduling rule is: give priority to the earliest available AGV to complete the material handling task.

[0188] Construct 6 actions to complete the selection and matching of processes and equipment, in which the selection of processes gives priority to the next process of the work that may be delayed:

[0189] Action 1: The selection rule for the process is SOT rule, and the selection rule for the equipment is rule M 1.

[0190] Action 2: The selection rule for the process is EDD rule, and the selection rule for the equipment is rule M 1.

[0191] Action 3: The selection rule for the process is STR / OP rule, and the selection rule for the equipment is rule M 1.

[0192] Action 4: The selection rule for the process is SOT rule, and the selection rule for the equipment is rule M 2.

[0193] Action 5: The selection rule for the process is EDD rule, and the selection rule for the equipment is rule M 2.

[0194] Action 6: The selection rule for the process is STR / OP rule, and the selection rule for the equipment is rule M 2.

[0195] in:

[0196] Device matching rule 1 (rule M 1): Select the machine with the smallest earliest available time (TC) at time t.

[0197] Device matching rule 2 (rule M 2): Select the machine with the lowest utilization (U) at time t.

[0198] The expected delay time for jobs that are expected to be delivered late:

[0199] DT=UD(j)+TC ave -Ddl(j)

[0200] Where TC ave is the average completion time of the current equipment, and UD(j) is the sum of the processing times of the unfinished processes of the remaining unfinished job j.

[0201] Step 4.3 Construct the reward function of S-Agent

[0202] At the initial moment, the scheduling system is in the initial state s0. The subsequent state is determined by s t Transfer to t+1 During the process, the agent selects an action a and a state value s t Transfer to t+1 And generate a reward value r t+1 Among them, the steps for setting the reward function are as follows:

[0203] [1] Determine whether the scheduling of all processes has been completed;

[0204] [2] If completed, calculate the total delay time Dletd t .

[0205] If Dletd t >1 and equipment utilization u a ≥0.6, reward value R t Return a ;

[0206] If Dletd t >1 and equipment utilization u a <0.6, reward value R t Return-Dletd t ;

[0207] If Dletd t ≤1 and the reward value Rt Return -ln(abs(Dletd t ));

[0208] [3] If not completed, the reward value is R t Returns 0.

[0209] Step 5. The steps to build the DQN algorithm model are as follows

[0210] Step 5.1 Build an Order Accepting DQN

[0211] Build the order acceptance DQN (A-DQN) to perceive the Q deep neural network structure (AQ-Net) of the production system state characteristics, including the input layer Hidden Layer Output Layer The number of input layer nodes is 8, corresponding to the state feature U of A-Agent O , Rp, Pr, nCost re ,nCost po ,nCost op , Dv O ,nCost op . The number of nodes in the output layer is 2, corresponding to the actions a1 and a2 of A-Agent. The hidden layer has 5 layers, each with 6 nodes, and the Adam optimization algorithm is used. The target neural network AT-Net is established with the same structure. The experience replay pool D with a capacity of 2000 is established A , storage experience (s t ,atd,r t ,s t+1 ) is used for experience playback.

[0212] Step 5.2 Build the Scheduling DQN

[0213] Construct a scheduling DQN (S-DQN) to perceive the state characteristics of the scheduling system. The Q deep neural network structure (SQ-Net) includes an input layer Hidden Layer Output Layer The number of input layer nodes is 5, corresponding to the state feature U of S-Agent ave , U std , CO ave , CO std , D act . The number of output layer nodes is 6, corresponding to S-Agent actions R1, R2, R3, R4, R5, R6. There are 5 hidden layers, each with 6 nodes, and the Adam optimization algorithm is used. The target neural network ST-Net is established with the same structure. An experience replay pool D with a capacity of 2000 is established S , storage experience (s t ,at d,r t ,s t+1 ) is used for experience playback.

[0214] Step 6. The steps for training the DQN algorithm model are as follows

[0215] Step 6.1 Extract the scheduling training set from the actual production system

[0216] Taking the actual order acceptance and scheduling process of a certain enterprise's intelligent manufacturing workshop as an example, 10 training set cases are extracted from the intelligent manufacturing workshop. Each case has 20 orders, each order contains 1-10 jobs, each order contains 1-2 processes, and the total number of processes is 100-150. There are 10 production equipment and 10 AGVs in each case. The same type of process can be performed on the corresponding machine, and the AGV equipment can perform any material handling task.

[0217] Step 6.2 Training Scheduled DQN (S-DQN)

[0218] [1] Set the number of training times L = 5000

[0219] [2] Start the i-th training (i=1 to L)

[0220] [3] The training case is the i-th case in the case set: Case = Cases[i], initialization state Its status characteristics

[0221] [4] Traverse all the processes in the case Case (t = 1 to d) and select action a s Match all processes and equipment.

[0222] Action search probability a e ←random(1)

[0223] action

[0224] [5] Update status parameters

[0225] [6] Update reward value

[0226] [7] Experience set increases experience D s .append(Ф,a,r,Ф t+1 )

[0227] [8] Arrival weight update step step up Update the weight θ′ of ST-Net

[0228] [9] Arrival experience playback step number step dr Real-time experience playback

[0229]

[10] Start the i+1th training

[0230] Step 6.3 Training Order Accepting DQN (A-CDQN)

[0231] [1] Set the number of training times Len = 2000

[0232] [2] Start the i-th training (i=1 to Len)

[0233] [3] The training case is the i-th case in the case set: Case OA =Cases OA [i], initialization state Its status characteristics

[0234] [4] Traverse all orders in the case Case (k = 1 to K)

[0235] [5] Update the production status through S-DQN Get the status characteristics under this production status Select action a A Decide whether to accept the order.

[0236] Action search probability a e ←random(1)

[0237] action

[0238] [6] Update status parameters

[0239] [7] Update reward value

[0240] [8] Experience Set Increases Experience

[0241] [9] Arrival weight update step step up Update the weight θ′ of AT-Net

[0242]

[10] Arrival experience playback step number step dr Real-time experience playback

[0243]

[11] Start the i+1th training.

[0244] This example also discloses a real-time order acceptance and scheduling system for a flexible job shop, including an input module and a decision module connected in sequence, wherein the input module is used to input characteristic attribute data of an order to be decided, and the decision module makes an implementation decision on the acceptance of the order and subsequent production scheduling according to the aforementioned method.

[0245] The following is a specific experimental example based on this system:

[0246] Experimental Example 1

[0247] Take the actual order acceptance and scheduling process in a smart manufacturing workshop of an enterprise as an example. The workshop has 10 production equipment and 10 AGVs. 100 cases of production logistics collaborative scheduling in the workshop are extracted, and the number of processes in each case is 100-150.

[0248] On the basis of the method proposed in this example, in order to verify the feasibility of the established scheduling agent, this experiment compares the scheduling agent used in this example with the traditional scheduling rules, applies it to the scheduling case set, and compares the solution quality of the scheduling agent used in the present invention and the traditional scheduling rules.

[0249] Figure 2 The results show that the scheduling agent proposed in this invention effectively improves the solution quality of the flexible job shop scheduling problem instance. Figure 2 Agent S It is the scheduling agent proposed in this invention; Rule1 sets the selection rule of the process as SOT rule and the selection rule of the equipment as rule M 1; Rule2 sets the process selection rule to EDD rule and the equipment selection rule to rule M 1; Rule3 sets the process selection rule to STR / OP rule and the equipment selection rule to rule M 1; Rule4 sets the process selection rule to SOT rule and the equipment selection rule to rule M 2; Rule5 sets the process selection rule to EDD rule and the equipment selection rule to rule M 2; Rule6 sets the process selection rule to STR / OP rule and the equipment selection rule to rule M 2.

[0250] Experimental Example 2:

[0251] Take the actual order acceptance and scheduling process of a smart manufacturing workshop of an enterprise as an example. The workshop has 10 production equipment and 10 AGVs. 100 cases of order acceptance and scheduling of the workshop are extracted. Each case has 20 orders, each order contains 1-10 jobs, each order contains 1-2 processes, and the total number of processes is 100-150.

[0252] On the basis of Example 1, in order to verify the feasibility of the multi-agent decision-making model and the corresponding DQN algorithm, this experiment applies the DQN-based multi-agent decision-making model and the general decision-making model adopted by the present invention to the order acceptance and scheduling case set, and compares the solution quality of the DQN-based multi-agent decision-making model and the general decision-making model adopted by the present invention.

[0253] Figure 3 The results show that the method for real-time order acceptance and scheduling in flexible job shops proposed in the present invention effectively improves the solution quality of the instance of real-time order acceptance and scheduling problem in flexible job shops. Figure 3 The multi-agent decision model in the formula is as follows: maxI represents the traditional decision model with the goal of maximizing profit; Random represents the random order acceptance model.

[0254] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions and / or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by the present invention shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A real-time order acceptance and scheduling method for a flexible job shop, characterized in that: These include: Step 1 extracts relevant data from the production site and builds a flexible job shop environment model based on the relevant data; Step 2 constructs an agent decision model, wherein the multi-agent decision model includes an order acceptance agent and a production logistics collaborative scheduling agent, which is used to simulate the execution process of order acceptance and scheduling in the production system; The order acceptance agent model includes state characteristics, actions, and reward functions; The production logistics collaborative scheduling agent model includes state characteristics, actions, and reward functions; Step 3: Based on the constructed order acceptance agent and production logistics collaborative scheduling agent, a corresponding DQN algorithm model is constructed, wherein the DQN algorithm model includes an order acceptance DQN and a scheduling DQN; Step 4: Based on the interaction with the agent, the order acceptance DQN and scheduling DQN algorithm models are trained. The training of the order acceptance DQN is completed on the basis of the scheduling DQN which has a better effect. Step 5 deploys the trained order acceptance DQN and scheduling DQN. The order acceptance DQN makes a decision on whether to accept the order, and the scheduling DQN makes a decision on the production scheduling of the order.

2. The method according to claim 1, characterized in that: In step 1, the relevant data includes order information and status, machine information and status, workpiece information and status, process information and status, and AGV information and status.

3. The method according to claim 2, characterized in that In the job shop environment model in step 1, the randomly arriving orders are represented as O = [O1, O2, ... n …O N ], Order O n The characteristic attributes of include its delivery deadline ddl, arrival time ontime, price price and delay penalty coefficient pu; Order O n All the tasks in the group form a task set Right now Indicates order O n The dth job in Contains a set of processes, and the process set is represented as All processes can be divided into K categories, and the process type corresponds to the equipment type one by one, that is, the type attribute k∈[1,2,…K] of the process or equipment, and the order O n The number of each type of process in is expressed as Indicates order O n The number of processes with internal process type k; The device set is represented by M = [M1, M2, ... M c …M C ],A=[A1,A2…A q …A Q ] represents the logistics task set, V = [V1, V2…V r …V R ] represents the AGV set; The called open production resource has a time window constraint, which is defined by the start time winStart and the end time winEnd; Order O n The acceptance decision is expressed as As shown below:

4. The method according to claim 3, characterized in that The construction of the multi-agent decision-making model follows the following strategy: Both the order acceptance decision and the real-time production logistics collaborative scheduling are modeled as Markov decision processes; In the order acceptance agent under open manufacturing resources, the decision point occurs when a new order arrives. When a new order arrives, the order acceptance agent decides whether to accept the order based on the production status characteristics provided by the production system and the characteristics of the new order, and determines the allocation of open production resources. The plan minimizes production delays, including estimated costs and predicted capacity gaps. The scheduling agent obtains production tasks based on the decision of the order acceptance agent and reschedules production and logistics tasks according to the current production status of the flexible job shop. The goal of the scheduling agent is to minimize the total delayed delivery time of all orders and set the corresponding reward function accordingly. The scheduling agent assigns all pending processes to production equipment and determines the estimated production processing time. During the production scheduling process, if a production process has a subsequent process, a material handling task is generated and assigned to the earliest available AGV to complete the logistics scheduling within the production flexible operation workshop; The scheduling agent completes the formulation of the scheduling plan and updates the scheduling plan to the production system. The production system executes according to the plan and transmits the current status of the production system to the order acceptance agent and the scheduling agent in real time when necessary.

5. The method according to claim 4, characterized in that The state characteristics of the order acceptance agent model include: The urgency of the order O , which represents the ratio of the processing time of all processes in an order to the remaining processing time; Expected revenue, Rp, represents the price of all accepted orders minus expected costs; The expected rate of return, Pr, represents the ratio of the expected total benefits to the expected total costs; Standardized production execution cost nCost re , which is the ratio of the expected maximum return to the production execution cost; Normalized maximum cost index nCost for calling open production resources po ; Normalized maximum delay cost index nCost op , represents the ratio of expected maximum revenue to expected deferred cost under basic capacity; The average production capacity gap of orders Dv O , represents the average difference between the production demand and the actual available production capacity for all types of work processes specified in the order; The normalized return index nEarnp represents the ratio of the expected maximum return to the average revenue of the accepted orders; The actions of the order acceptance agent model include: Action 1: Do not accept orders and do not call open production resources; Action 2: Accept the order and call on external manufacturing resources, with the goal of minimizing the estimated cost; The state transitions of the order acceptance agent model include: At the initial time t0, the production system is in the initial state s0, at which time the arrival status of all orders is not arrived; the subsequent status is s k Transfer to k+1 During the process, time is t k Transfer to k+1 , the agent selects an action a for the kth order and decides whether to accept the order. The state value is s k Transfer to k+1 And generate a reward value r k+1 , until the state is transferred to the terminal state t=t0+T to obtain the acceptance decision of all orders in period T; The order-taking agent model uses the normalized expected total revenue over the cycle time as a reward value.

6. The method according to claim 5, characterized in that The state characteristics of the scheduling agent model include: Average machine utilization U ave ; Standard deviation of machine utilization U std ; Average order completion rate CO ave ; The standard deviation of the completion rate of each order process CO std ; The actual delay time of the order D act ; In the scheduling agent model, the selection criteria for production equipment are defined by the following two rules: Device matching rule M 1: Select the machine with the smallest earliest available time TC at time t; Device matching rule M 2: Select the machine with the lowest utilization U at time t; The actions of the scheduling agent model include: Action 1: Prioritize the next unscheduled process of the workpiece with the largest delay time, otherwise, the process selection is determined according to the SOT rule. In the selection of equipment, the machine with the earliest available time is given priority, that is, rule M 1; Action 2: Prioritize the next unscheduled process of the workpiece with the largest delay time. Otherwise, the process selection follows the earliest delivery time rule. The equipment selection rule is rule M 1; Action 3: Prioritize the next unscheduled process of the workpiece with the largest delay time. Otherwise, the process selection follows the STR / OP rule, and the equipment selection rule follows rule M 1; For actions 4, 5, and 6, the process selection rules refer to SOT, EDD, and STR / OP respectively, and the equipment selection rules are all rule M 2; The state transition of the scheduling agent model includes: At the initial moment, the scheduling system is in the initial state s0, at which time the scheduling state of all processes is unscheduled, and all machines are in the idle state. t Transfer to t+1 During the process, the agent selects an action a, matches a device with a process, and the state value s t Transfer to t+1 And generate a reward value r t+1 , until the state is transferred to the terminal state to obtain the scheduling plan for all processes. The scheduling agent model uses a logarithmic function to minimize the delivery time delay as the basis for defining the reward function; at the same time, equipment utilization is used as an auxiliary indicator to consider providing positive rewards for scenarios where the delay time is extended but the equipment load is high.

7. The method according to claim 6, characterized in that The construction of the DQN algorithm model follows the following strategy: The construction of the DQN model includes two parts: order acceptance DQN model construction and scheduling DQN model construction. In the order acceptance DQN, the goal of the model is to determine the order acceptance combination that arrives in the cycle to obtain a better overall return; In the scheduling DQN, the decision goal is to match production equipment for each production task to minimize the production delay time; The specific construction steps of the two DQNs are as follows: Construct the Q deep neural network structure of order acceptance DQN to perceive the state characteristics of the production system, including input layer, hidden layer, and output layer; the number of input layer nodes corresponds to the number of state characteristics of the order acceptance agent model, and the number of output layer nodes corresponds to the number of actions of the order acceptance agent model. Adopt the Adam optimization algorithm to establish the target neural network with the same structure, establish an experience replay pool D with a capacity of N, and store the experience (s t ,atd,r t ,s t+1 ) is used for experience playback; Construct a Q deep neural network structure for the scheduling DQN to perceive the state characteristics of the scheduling system, including an input layer, a hidden layer, and an output layer; the number of input layer nodes corresponds to the number of state characteristics of the scheduling agent model, and the number of output layer nodes corresponds to the number of actions of the scheduling agent model, using the Adam optimization algorithm; The target neural network is established with the same structure, and an experience replay pool D with a capacity of N is established to store experience (s t ,a t d,r t ,s t+1 ) is used for experience playback; In the production system, when a new order arrives, the order acceptance DQN perceives the production status characteristics of the production system and makes an order acceptance decision. The scheduling DQN perceives the current state of the scheduling system and reschedules the processes of the order for which the acceptance decision is made and the processes that are not produced in the current workshop. The state transition process of the production system is jointly affected by the decisions made by the order acceptance DQN and the scheduling DQN.

8. The method according to claim 7, characterized in that In step 4, the steps for training the DQN algorithm model are as follows: In the training process of the DQN model, the scheduling DQN is first trained to ensure that the DQN can produce a scheduling solution with a better delay time; The scheduling training set is input into the scheduling DQN model for training. Each training is conducted on one case in the training set, and each training traverses all the production tasks in the case for scheduling. As the training progresses, the Q network is updated based on the target value, and the Q neural network is updated to the target neural network every n steps. The training gradually converges, the degree of exploration slowly decreases, and the probability of selecting the exploration action gradually decreases as the greed coefficient e decreases. Based on the trained production logistics collaborative scheduling DQN, the order acceptance decision DQN is trained; The order acceptance training set is input into the order acceptance DQN model for training. Each training session trains one case in the training set. Each training session traverses all orders in the case to make acceptance decisions. The state value of the order acceptance decision is provided by the production environment. The state value of the production environment changes with the progress of time steps according to the latest scheduling plan of the scheduling DQN. As the training progresses, the Q network is updated based on the target value. The Q neural network is updated to the target neural network every n steps. The training gradually converges, the degree of exploration slowly decreases, and the probability of selecting the exploration action gradually decreases with the decrease of the greed coefficient e.

9. A real-time order acceptance and scheduling system for a flexible job shop, characterized in that: It comprises an input module and a decision module connected in sequence, wherein the input module is used to input characteristic attribute data of an order to be decided, and the decision module makes a decision on the acceptance of the order and subsequent production scheduling according to any method described in claims 1-8.

Images