Dynamic cloud order acceptance and scheduling method based on deep reinforcement learning
By adopting deep reinforcement learning methods in the cloud manufacturing environment, designing order acceptance and order placing agents, and collaborating on optimizing the scheduling of cloud orders and own orders, the problems of order uncertainty and production disturbance in cloud manufacturing are solved, and the problems of order uncertainty and production disturbance in cloud manufacturing are improved. The profit and resource utilization rate of the enterprise.
Patent Information
- Application Number
- CN202111085349.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the cloud manufacturing environment, how enterprises can coordinately manage their own orders and cloud orders, optimize production resource allocation, and increase profits, especially when facing challenges such as order uncertainty, workshop production disturbances and lead time sensitivity, The technology lacks effective dynamic scheduling strategies.
Using the dynamic acceptance and scheduling method of cloud orders based on deep reinforcement learning, the order reception agent and the order placing agent are designed, with the maximum profit, minimum drag time and minimum disturbance respectively, and the deep Q-value network (DQN) algorithm coordinated. Decision-making, combining workshop information and order characteristics, optimize order acceptance and scheduling.
It improves the overall profit and production capacity utilization rate of the enterprise, effectively responds to dynamically arrived cloud orders, reduces order delays and production disturbances, and improves the overall efficiency of workshop resources.
Smart Images

Figure CN113935586B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a cloud order dynamic acceptance and scheduling method based on deep reinforcement learning. Background Art
[0002] Cloud manufacturing is a new intelligent manufacturing model that uses the Internet and service platforms to provide various on-demand manufacturing services for customer needs. Cloud manufacturing has the characteristics of "centralized use of decentralized resources" and "centralized resources for decentralized services", providing manufacturing capabilities for enterprises and customers and realizing resource sharing. After receiving the manufacturing task, the cloud platform looks for production enterprises that meet user needs. Only efficient service resource allocation methods can ensure the promotion and application of the cloud manufacturing model. In the cloud manufacturing environment, how enterprises allocate resources, adapt to dynamically arriving orders, and increase their own profits has become one of the key core issues that enterprises are concerned about when participating in the cloud manufacturing model. Most of the existing research focuses on the architecture, functions, and technical implementation of cloud manufacturing platforms. There is still a lack of research on the production scheduling problem of enterprise workshops under the background of cloud manufacturing.
[0003] The orders of production enterprises participating in cloud manufacturing can be divided into two categories: their own orders and cloud orders. Their own orders can be considered certain, while cloud orders have extremely high uncertainty. Customers submit orders to the cloud platform, and the platform assigns the orders to the appropriate enterprises. The enterprises make decisions on whether to accept the orders and arrange the production of the received orders. After receiving the cloud orders, the enterprises arrange the best scheduling plan for all orders based on their own remaining capacity to maximize profits. Therefore, how enterprises in the cloud manufacturing environment can coordinate the production of the two types of orders and reasonably schedule and optimize the allocation of production resources of enterprises is of great significance to improving the profits of enterprises themselves, and is also a key issue that enterprises need to solve after participating in cloud manufacturing.
[0004] In recent years, good research results have been achieved on the problem of order acceptance and scheduling. According to the insertion method of cloud orders, this type of problem can be divided into two categories: static scheduling and dynamic scheduling. Static scheduling means that cloud orders will not change the production of own orders, and dynamic scheduling means that cloud orders are rescheduled with the initial scheduling plan before production starts. The research on order acceptance and scheduling problems in dynamic environments faces the following challenges:
[0005] (1) Order uncertainty. The uncertainty of order arrival time, processing quantity, delivery date, price and other information makes it difficult for enterprises to make decisions on current orders based on the arrival of future orders. For example, after an enterprise receives a cloud order A with a smaller profit, a cloud order B with a larger profit arrives. However, after receiving order A, the workshop is no longer able to receive order B, resulting in a loss of part of the profit. If order A is rejected, it may result in a long wait for high-profit orders, resulting in a waste of enterprise resources. In a parallel machining environment, Emmami et al. considered the integrated optimization problem of order acceptance and scheduling under the condition of uncertainty of information such as order delivery date, revenue, delay penalty and required installation time. Wang et al. studied the order acceptance and sorting problem of a dual-machine assembly line workshop before the enterprise planning period under the condition of uncertainty of order price, processing time, delivery date and other information, and proposed an improved heuristic algorithm and branch-and-bound algorithm.
[0006] (2) Disturbance to workshop production. Before the cloud orders arrive, the company's own orders have already generated a scheduling plan. For dynamic scheduling problems, receiving cloud orders will disrupt the original scheduling plan of the workshop. Changes in personnel, machinery and equipment, and materials will increase the difficulty of managing the company's production. Moratori et al. studied the insertion of new orders into the current schedule and proposed a new effective matching strategy to maintain the effectiveness of scheduling and the stability of workshop production. Yan Haoyun et al. established a mixed integer programming model for the order rescheduling problem and adopted a heuristic solution method based on dynamic programming ideas to effectively solve the problem of inserting urgent orders.
[0007] (3) Sensitivity to delivery dates. When an order is not delivered on time, not only will it incur delay costs, but it will also damage the company's reputation. Therefore, when accepting an order, the company needs to fully consider whether the remaining production capacity can complete the order within the delivery period. Slotnick et al. studied the order acceptance decision problem when production capacity is limited, orders are discounted due to delayed delivery, but early delivery is not punished or rewarded, and solved it using a heuristic algorithm based on the relaxation method. Cesaret et al. proposed a taboo search algorithm to solve the single-machine order acceptance and sorting problem considering delay penalties. Xiao et al. used a partially optimized heuristic simulated annealing algorithm to solve the order reception and weighted delay arrangement assembly line operation scheduling problem faced by enterprises on assembly line production lines.
[0008] In recent years, many studies have comprehensively considered factors such as the production capacity of the enterprise workshop, the profit of the order, and the production cost of the order, and finally determined the order acceptance and scheduling strategy. Song Lijun et al. proposed a decision-making method for the integration of order acceptance and scheduling in assembly line production enterprises, and constructed a heuristic algorithm solution model based on the simulated annealing process. et al. studied the fuzzy order decision and sequencing problem of equivalent parallel machine workshops and solved it by using an improved genetic algorithm. Tarhan et al. proposed a new iterative local search algorithm to solve the order acceptance and scheduling problem. B. Vahedi-Nouri et al. established a multi-objective model that comprehensively considers the utility and fairness of customers and factories, and used the Epsilon constraint method to decide whether the cloud platform accepts orders, determine the allocation of factories and machines, and order prices. Wang et al. proposed a variable particle swarm optimization VDPSO algorithm to solve the cloud order acceptance problem in order to utilize the idle time period of the enterprise processing. After accepting the order, the Giffler-Thompson algorithm was used to schedule the order. Li et al. considered the situation where a manufacturer received multiple orders and proposed three exact algorithms to determine the best set of received orders and the corresponding production plan. Wu et al. proposed an improved water flow algorithm to solve the order acceptance problem and parallel machine scheduling problem. Wang Zhen et al. used an improved second-order particle swarm algorithm to solve the optimal scheduling sequence of workpieces in order to solve the problem of residual capacity utilization of enterprises. Heuristic algorithms have the disadvantages of needing to reconstruct feasible solutions for dynamic scheduling problems, and cannot be applied to the problem of dynamic arrival of cloud orders. Artificial intelligence algorithms need to be further applied.
[0009] At present, a class of artificial intelligence algorithms such as reinforcement learning have been successfully applied to the fields of enterprise order acceptance, workshop scheduling, logistics management, etc., and have received great attention from all walks of life at home and abroad. Wang Xiaohuan et al. used the SMART algorithm to solve the uncertainty problem of enterprises in the order acceptance decision-making process. Hao Juan et al. used the average reinforcement learning algorithm to study the order acceptance problem of enterprises in the order production mode under uncertain environment. Li et al. aimed at the problem of increased demand and limited resources, and established a semi-Markov decision problem by setting prices and lead times to influence demand, and used the Q-learning algorithm to decide whether to accept or reject orders. Since orders arrive dynamically, the production information of the workshop is also dynamically changing. The above literature did not formulate order acceptance strategies based on the dynamic information of the workshop, and lacked a basis for decision-making. For the workshop scheduling problem, many studies abstracted it into a sequential decision problem and solved it using the reinforcement learning algorithm. He Junjie et al. proposed a multi-agent loop proximal strategy optimization algorithm for the dynamic scheduling problem of textile fabric dyeing workshops with dynamic arrival of tasks, with the goal of minimizing the total delay time. Chen Yong et al. established a multi-disturbance workshop flexible scheduling model based on cellular machines and reinforcement learning algorithms to address the problem of large-scale equipment manufacturing enterprises with many disturbances and large impacts. Xia Jin et al. proposed a Q-learning algorithm with dynamically changing learning rate and exploration rate to optimize production re-decision problems. Xiao Pengfei et al. proposed a deep reinforcement learning algorithm based on temporal difference to solve the non-permutation assembly line scheduling problem. Wang Ling et al. proposed a deep reinforcement learning and iterative greedy algorithm to solve assembly line scheduling with the goal of minimizing the maximum completion time. Ren Jianfeng et al. used the long short-term memory network model as the main function and embedded a pointer network in the model to solve the job shop scheduling problem. However, none of the above literatures involve the scheduling problem of hybrid assembly line workshops. In fact, since the dispatched cloud orders have the characteristics of small batches and multiple varieties, most of the workshops involved in cloud manufacturing are hybrid assembly lines. Based on this, the present invention solves the order reception and workshop scheduling problems of dynamically arriving cloud orders under the study of hybrid assembly line workshops. Summary of the invention
[0010] In response to the above-mentioned problems existing in the prior art, the present invention provides a cloud order dynamic acceptance and scheduling method based on deep reinforcement learning.
[0011] The present invention adopts the following technical scheme:
[0012] The cloud order dynamic acceptance and scheduling method based on deep reinforcement learning is as follows:
[0013] Step 1: The order-taking agent formulates an order acceptance strategy based on the information feedback from the workshop with the goal of maximizing profit;
[0014] Step 2: When the cloud order is accepted, the order scheduling agent will reschedule the remaining processes and cloud orders at the current moment with the minimum delay time and minimum disturbance as the integration goal; after the scheduling is completed, the production capacity of the workshop will be fed back to the order-taking agent.
[0015] Preferably, in step 1, the objective function of profit is as follows:
[0016]
[0017]
[0018] Among them, p b ,v,d b 、F b 、c b denote the price of the bth (b=1,2,…g) cloud order, the penalty factor of the unit delay time, the delay time, the fixed production cost, and the machine production cost respectively; p Q ,d Q 、F Q 、c Q denote the price, delay time, fixed production cost, and machine production cost of the Qth (Q=1, 2, ...h)th own order respectively;
[0019] In workshop scheduling, the minimum objective function for the cumulative delay time of all orders is:
[0020]
[0021] The objective function for minimizing workshop production disturbance is the cumulative difference in process time between the new and old scheduling schemes:
[0022]
[0023] In the formula, It represents the difference in start processing time between the new and old scheduling plans for the wth (w=1,2,…u)th process of the zth (z=1,2,…L)th order. It represents the difference in the end processing time between the new scheduling plan and the old scheduling plan for the w-th process of the z-th order.
[0024] Preferably, in step 1, the state space of the order-taking agent is the processing end time O of each machine. i (1≤i≤m), the cumulative processing time T of each machine i 1 (1≤i≤m), the number of cloud order workpieces n b 、Workpiece type type(b)、Delivery date D b 、Price b ;
[0025] In step 2, the state space of the order-scheduling agent is the processing end time O of each machine. i (1≤i≤m), the cumulative processing time T of each machine i 1 (1≤i≤m), the latest processing end time of each workpiece Remaining process E for each workpiece e (1≤e≤n), the processing schedule JM of each workpiece type on the machine.
[0026] Preferably, in step 1, the scheduling actions of the order-taking agent are accepting orders and rejecting orders, which are defined as:
[0027]
[0028] In step 2, the order scheduling agent first selects the rules for processing workpieces and machines at the same time, then selects the workpiece to be processed from the process candidate set according to the rules, and then selects the processing machine from the machine candidate set.
[0029] Preferably, in step 1, the reward function of the order-taking agent is
[0030] The order-taking agent takes profit maximization as its objective function. Each order has a corresponding price, fixed cost, machine processing cost, and delay cost. The order-taking agent needs to determine whether there is profit in accepting cloud orders based on the status of the workshop. The reward function is designed as follows:
[0031]
[0032]
[0033] In step 2, the reward function of the queue agent is
[0034] After accepting an order, when the order scheduling agent inserts a new order into production, it will affect the production schedule of the existing orders. It not only needs to consider the order delay problem, but also the disturbance problem of the newly inserted order. Therefore, the order scheduling agent takes the dual objectives of minimum order delay and minimum order insertion disturbance as the objective function.
[0035]
[0036]
[0037] In the above formula, the calculation method of the delay reward for each action is that when the workpiece is not completed, the delay time is calculated based on the latest start time of the process; when the workpiece is completed, the delay time is calculated based on the delivery date of the workpiece; where S is the cumulative total number of orders of the own orders and cloud orders, and r1 represents the order-taking agent at the decision time t k -1 transfer to tk The reward value; it is proved that minimizing the order delay time is equal to maximizing the reward function
[0038]
[0039] In the above formula, t j represents the extension time of the jth (j=1,2,…S) order;
[0040] When inserting cloud orders into production, it will cause changes in machinery, equipment, personnel, and materials. Therefore, when making decisions, it is expected that the overall arrangement of the order will be as different from the previous production as possible. The overall difference is calculated by the difference between the start time and the end time of each process of each workpiece, and the time difference of the process production is fed back to the order scheduling agent as a reward;
[0041]
[0042] Convert the single-step reward into the overall objective function:
[0043]
[0044] The smaller the difference between the order production arrangement of the new scheduling plan and the old scheduling plan, the smaller the disturbance in the workshop. Minimizing the production disturbance of order insertion is equal to maximizing the reward function.
[0045] When a new order comes in and the order-scheduling agent makes a new scheduling arrangement, the final reward will weigh the integrated reward functions r1 and r2. The weighted method is used to integrate the two goals, and the reward function needs to be normalized first. The method is as follows:
[0046]
[0047] Since the maximum value of the delay reward function and the maximum value of the order disturbance function are unknown, they are set as: the delay reward function value r1 max =0, Order perturbation function value Among them, C1 and C2 are small negative numbers and are smaller than all reward function values;
[0048] Convert the two objectives into a single objective optimization problem:
[0049]
[0050] In the above formula, w e represents the weight of the e(e=1,2)th target, r′ e Represents the normalized value of the e-th target value.
[0051] In the cloud manufacturing environment, in addition to processing and completing their own orders, enterprises can also accept cloud orders assigned by the cloud platform, thereby improving the utilization rate of their own resources and total profits. Aiming at the order acceptance and scheduling problem in the existing cloud order dynamic arrival environment, the present invention proposes a cloud order dynamic acceptance and scheduling method based on deep reinforcement learning, involving an improved deep Q value network (improved DQN) algorithm. The dynamic scheduling problem involved in the present invention includes two sub-problems: order acceptance and order scheduling, and an order scheduling agent and an order acceptance agent are designed respectively; wherein, the order acceptance agent aims to maximize profit, and the order scheduling agent aims to minimize delay time and minimize disturbance. When making decisions, the order acceptance agent comprehensively considers the order information assigned by the platform and the production status of the workshop, and then feeds back the order acceptance situation to the order scheduling agent. In the order scheduling agent, improved strategies such as the process candidate set with the latest start time in the critical path, the machine candidate set, and the earliest start of the workpiece on the scheduled machine are introduced to improve the coordination ability of the order scheduling agent for dynamically arriving cloud orders and the production of own orders. In addition, the present invention designs a dynamic interaction mechanism between the order-taking agent and the order-arranging agent. The order-arranging agent feeds back the workshop information to the order-taking agent, and the order-taking agent transmits the new order information to the order-arranging agent. The shared information provides a better basis for the agent to make decisions. Finally, based on the mixed flow shop processing environment, the present invention designs three groups of numerical simulation experiments of different scales. The results show that the improved DQN method proposed in the present invention is superior to the classic SPT rule, DQN, and Q-Learing algorithm, and can effectively improve the overall profit of the enterprise and the utilization rate of production capacity. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is based on the DQN reinforcement learning agent and the workshop interaction diagram.
[0053] Figure 2 is the agent interaction graph.
[0054] Figure 3 It is the best solution Gantt chart for cloud order quantity 5.
[0055] Figure 4 It is the best solution Gantt chart for cloud order quantity of 10.
[0056] Figure 5 It is the best solution Gantt chart for cloud order quantity of 20.
[0057] Figure 6 This is a comparison chart of average profits under different delay penalty factors.
[0058] Figure 7 This is a comparison chart of maximum profits under different delay penalty factors. DETAILED DESCRIPTION
[0059] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0060] Based on the existing MTO enterprise order decision research and DQN reinforcement learning algorithm, the present invention designs an order-taking agent and an order-scheduling agent method for the order decision problem of dynamically arriving cloud orders. The order-taking agent formulates an order acceptance strategy based on the information feedback from the workshop with the goal of maximizing profit. When the cloud order is accepted, the order-scheduling agent reschedules the remaining processes and cloud orders at the current moment with the minimum delay time and minimum disturbance as the integration goals. The order-scheduling agent combines the candidate set of processes with the latest start time and the candidate set of idle machines at the current moment, as well as improvement strategies such as arranging the earliest start of processing on the machine for the current process, to improve the efficiency of process scheduling. After the scheduling is completed, the production capacity of the workshop is fed back to the order-taking agent, and an efficient scheduling strategy is established through interactive training between the order-taking agent and the order-scheduling agent.
[0061] 1. Background description
[0062] The processing background of the enterprise in this embodiment is a mixed flow production workshop, which can be described as having m machines and n workpieces to be processed. Each workpiece contains w processing procedures, which can be 1 or more procedures, and the order of the procedures is fixed. Each procedure can be processed on all machines, and the processing time and processing cost are different. The workpiece can only choose one device for each procedure for processing, and all order delays will incur delay costs. The workshop needs to arrange the processing machine and start processing time for each procedure of the workpiece, so that the total delay of all orders is small and the production arrangement disturbance is small.
[0063] The following assumptions are made about the background:
[0064] (1) All cloud orders arrive dynamically and follow a Poisson distribution.
[0065] (2) The customer does not have any special circumstances such as canceling the order or changing the order information.
[0066] (3) When an order exceeds the delivery date, delay costs will be incurred.
[0067] (4) A machine can only process one workpiece at any time, and processing cannot be interrupted midway.
[0068] (5) There is no sequence constraint for the processing of different workpieces, but there is a sequence constraint for the processing of the same workpiece.
[0069] Under the premise of producing its own orders, an enterprise needs to make a choice of orders from the cloud manufacturing platform and complete the production arrangement. The order arrival follows a Poisson distribution with parameter λ, and each order includes the order quantity, price, delivery date, processing type, etc. Assuming that there are g cloud orders dynamically dispatched to an enterprise for processing, the enterprise needs to make a decision on whether to accept the current order based on the production information of the cloud order, the production situation of the workshop, and the arrival of future orders, so as to maximize the total profit of the enterprise (Formula 1).
[0070] If you choose to accept the cloud order, the accepted cloud order and the workshop production that has not started at the current moment will be rearranged, with the minimum total delay (Formula 3) and the minimum difference in workpiece production time before and after the cloud order is inserted (Formula 4) as the optimization goals. The workshop needs to arrange the processing machine and start processing time for each process of the workpiece, so that the total delay of all orders is small and the production arrangement disturbance is small. In summary, the decision variables of the present invention are whether to accept the cloud order, determine the workpiece processing machine and start processing time.
[0071] The total profit objective function of the enterprise is as follows:
[0072]
[0073]
[0074] Among them, p b ,v,d b 、F b 、c b They represent the price of the bth (b=1, 2, ...g) cloud order, the penalty factor of the unit delay time, the delay time, the fixed production cost, and the machine production cost. Q ,d Q 、F Q 、c Q They represent the price, delay time, fixed production cost, and machine production cost of the Qth (Q=1, 2, ...h) own order respectively.
[0075] In the workshop scheduling problem, the minimum objective function for the cumulative delay time of all orders is:
[0076]
[0077] The objective function for minimizing workshop production disturbance is the cumulative difference in process time between the new and old scheduling schemes:
[0078]
[0079] In the formula, It represents the difference in start processing time between the new and old scheduling plans for the wth (w=1,2,…u)th process of the zth (z=1,2,…L)th order. It represents the difference in the end processing time between the new scheduling plan and the old scheduling plan for the w-th process of the z-th order.
[0080] 2. Order dynamic scheduling method based on DQN reinforcement learning
[0081] The order dynamic scheduling problem of the present invention can be decomposed into two sub-problems: order acceptance and order scheduling. Order acceptance is to decide whether to accept cloud orders, and order scheduling is to allocate the accepted cloud orders and own orders to the appropriate machines for production and start processing time. As mentioned above, cloud orders are constantly arriving, and the production status of the workshop is always in dynamic change, which converts the cloud order dynamic scheduling problem into a sequential decision-making problem involving order acceptance and order scheduling. To this end, the present invention converts the problem into a reinforcement learning problem, and designs an order-accepting agent and an order scheduling agent to solve the two sub-problems respectively. The order-accepting agent aims to maximize profits, and the order scheduling agent aims to minimize delays and minimize order insertion production differences. For the order scheduling agent, scheduling strategies such as the candidate sets of workpieces and machine processing and the earliest start of processing improvements for the workpiece on the selected machine are designed.
[0082] 2.1DQN reinforcement learning algorithm
[0083] The present invention uses the Markov decision process G(S, A, P, R, γ) to describe the cloud order dynamic scheduling problem, which includes the state space S, the action space A, the state transition probability P, the reward function R, and the discount factor γ. Figure 1 shown.
[0084] 2.1.1 Agent
[0085] The present invention includes an order-taking agent and an order-scheduling agent. Since the objective functions of the two agents are different and they are non-homogeneous agents, two DQNs are used to represent the two agents respectively. The order-taking agent makes decisions on cloud orders based on the workshop status and order information fed back by the order-scheduling agent. After accepting the cloud order, the cloud order information is transmitted to the order-scheduling agent. After the order-scheduling agent arranges the production of the cloud order, it feeds back the workshop status to the order-taking agent, providing a basis for whether to accept the next order. Figure 2 shown.
[0086] 2.1.2 State Space
[0087] The state space of the order-taking agent is the processing end time O of each machine. i (1≤i≤m), the cumulative processing time T of each machine i 1 (1≤i≤m), the number of cloud order workpieces n b 、Workpiece type type(b)、Delivery date D b 、Price b .
[0088] The state space of the order scheduling agent is the processing end time O of each machine. i (1≤i≤m), the cumulative processing time T of each machine i 1 (1≤i≤m), the latest processing end time of each workpiece Remaining process E for each workpiece e (1≤e≤n), the processing schedule JM of each workpiece type on the machine.
[0089] 2.1.3 Action Space
[0090] The order dynamic scheduling decision space is a set of scheduling decisions of intelligent agents when the order is in different states, mainly including the order receiving scheduling decision space and the order arranging scheduling decision space.
[0091] 1) The scheduling actions of the order-taking agent are accepting orders and rejecting orders, which are defined as:
[0092]
[0093] 2) The order scheduling agent first selects the rules for processing workpieces and machines at the same time, then selects the workpiece to be processed from the process candidate set according to the rules, and then selects the processing machine from the machine candidate set. Its neural network structure is shown in the following table.
[0094] Table 1 Neural network structure of the queuing agent
[0095] enter Output Hidden layer 1: fully connected layer (None, state dimension) (None, 1600) Hidden layer 2: Convolutional layer (None, 1600) (None, 20, 20, 8) Hidden layer 3: Pooling layer (None, 20, 20, 8) (None, 10, 10, 8) Hidden layer 4: Convolutional layer (None, 10, 10, 8) (None, 5, 5, 16) Hidden layer 5: Convolutional layer (None, 5, 5, 16) (None, 5, 5, 32) Hidden layer 6: fully connected layer (None, 5, 5, 32) (None, 512) Output layer 1 (None, 512) (None, 10) Output layer 2 (None, 512) (None, 5)
[0096] The last 10 neurons outputted from output layer 1 represent 10 rules for selecting processing procedures, and the 5 neurons in output layer 2 represent 5 rules for selecting processing machines. One rule is selected from output layer 1 and output layer 2 each time. The rules are shown in Tables 2 and 3 below.
[0097] Table 2 Candidate behavior set of artifacts
[0098]
[0099]
[0100] The remaining processing time of the workpiece is the sum of the maximum processing time of the remaining processes on each machine, and the maximum processing time of the next process is equal to the longest processing time of this process on all machines.
[0101] Table 3. Machine candidate behavior set
[0102] Serial number describe 1 Select the machine with the shortest processing time, SPT rule 2 Select the machine with the lowest utilization rate at the current moment 3 Select the processing machine with the lowest process cost, SPC rules 4 Choose the machine with the shortest waiting time for the process 5 Choose the machine that can complete the process first
[0103] 2.1.4 Rewards
[0104] 1) Reward function of the order-taking agent
[0105] The order-taking agent uses profit maximization as its objective function. Each order has a corresponding price, fixed cost, machine processing cost, and delay cost. The order-taking agent needs to determine whether there is profit in accepting cloud orders based on the status of the workshop. The reward function is designed as follows:
[0106]
[0107]
[0108] 2) Reward function of the queue agent
[0109] After accepting an order, when the order scheduling agent inserts a new order into production, it will affect the production schedule of the existing order. It is necessary to consider not only the order delay problem, but also the disturbance problem of the newly inserted order. Therefore, the order scheduling agent takes the dual objectives of minimum order delay and minimum order insertion disturbance as the objective function.
[0110]
[0111]
[0112] The calculation method of the delay reward for each action in the above formula is that when the workpiece is not completed, the delay time is calculated based on the latest start time of the process. When the workpiece is completed, the delay time is calculated based on the delivery date of the workpiece. In the formula, S is the cumulative total number of orders of the own orders and cloud orders, and r1 represents the order-taking agent at the decision time t k -1 transfer to t k It can be proved that minimizing the order delay time is equal to maximizing the reward function
[0113]
[0114] In the above formula, t j Indicates the extension time of the jth (j=1,2,…S) order.
[0115] When inserting a cloud order into production, it will cause changes in machinery, equipment, personnel, materials, etc. Therefore, when making decisions, it is expected that the overall arrangement of the order will be as different from the previous production as possible. The overall difference is calculated by the difference between the start time and the end time of each process of each workpiece, and the time difference of the process production is fed back to the order scheduling agent as a reward.
[0116]
[0117] Convert the single-step reward into the overall objective function:
[0118]
[0119] Obviously, the smaller the difference between the order production arrangement of the new scheduling plan and the old scheduling plan, the smaller the workshop disturbance will be. It can be obtained that minimizing the production disturbance of order insertion is equal to maximizing the reward function
[0120] When a new order comes in and the order-scheduling agent makes a new scheduling arrangement, the final reward will weigh the integrated reward functions r1 and r2. The present invention uses a weighted method to integrate the two goals, and the reward function needs to be normalized first, as follows:
[0121]
[0122] In the present invention, since the maximum value of the delay reward function and the maximum value of the order disturbance function are unknown, they are set as follows: the delay reward function value r1 max =0, Order perturbation function value Among them, C1 and C2 are small negative numbers and are smaller than all reward function values.
[0123] Convert the two objectives into a single objective optimization problem:
[0124]
[0125] In the above formula, w e represents the weight of the e(e=1,2)th target, r′ e Represents the value obtained after normalization of the e-th target value. As for the determination of weights, since enterprises prefer that the production of orders will not be delayed rather than production disturbances, the weight of order delay target optimization is set slightly higher than that of order production disturbances in the present invention.
[0126] 2.1.5 Improvement of cloud order dynamic scheduling strategy
[0127] The delay of workpieces not only affects the service quality, but also causes the delay cost. If the workpieces with earlier delivery dates can be processed first and the workpieces with later delivery dates can be postponed for production, the delay cost will be reduced. The present invention determines the processing candidate set at the current moment according to the latest start processing time of each workpiece. When the workpiece selects a processing machine, if the earliest processing time of the workpiece is less than the earliest processing time of the machine, the workpiece will be caused to wait. Therefore, choosing the currently idle machine can avoid waiting. The specific improvement strategy is as follows.
[0128] 1) Construction of candidate process sets combined with critical paths. First, record the processing time required for each process of each workpiece. Since it can be processed on multiple machines with different processing times, the average processing time is used as the time required to process the process. When the cumulative processing time of all processes of a workpiece exceeds the delivery date, the workpiece needs to be processed immediately. The time pace of machine processing is inconsistent. The present invention uses the latest processing end time of all machines as the time standard. For example, the delivery date of a workpiece is 15, the average processing time of the first process is 5.2, the second is 5, and the third is 6.1, as follows:
[0129] Table 4 First calculation
[0130]
[0131]
[0132] The cumulative processing time of the workpiece is 16.3, which exceeds the delivery date, so the latest processing time of process 1 is 0 time, process 2 is 5 time, and process 3 is 10 time. In other words, when processing starts at 0 time, in order to avoid delay, process 1 is selected into the process candidate set.
[0133] Table 5 Second calculation
[0134] Processing time Latest start time of processing Process 1: Processed 2 — Process 2: Unprocessed 5 3 Process 3: Unprocessed 6.1 8
[0135] At this time, the cumulative processing time of the workpiece is 13.1, which is within the delivery period. The latest start time of process 2 and process 3 is 3.9 and 8.9, respectively. Since the time can only be advanced but not postponed, time 3 is taken as the latest start time of process 2. When the time is greater than or equal to 3 times, process 2 has not been processed yet, and it is selected into the process candidate set.
[0136] 2) Construction of the machine candidate set. The machine candidate set is first selected from the idle machines at the current moment, and the processing time needs to be less than the average processing time. If all machines are busy at the current moment, the machine candidate set becomes the machine whose processing time is less than the average processing time.
[0137] 3) Determination of the start processing time of the workpiece. After the scheduling agent selects the workpiece and the machine, it arranges the workpiece for processing on the machine. The start processing time of the workpiece is the earliest time that the machine can process.
[0138] 2.2 Model Update
[0139] In combination with the characteristics of the research problem, this paper proposes an improved DQN algorithm, which includes two algorithms. The first algorithm is the order scheduling agent, which is mainly used to determine the processing machine and start processing time of the unprocessed process at the current moment. The second algorithm is the order receiving agent, which determines whether the cloud order can be accepted according to the current workshop status. The algorithm framework is as follows:
[0140]
[0141]
[0142]
[0143] 3 Experimental demonstration
[0144] Python 3.7 was used for programming, tensorflow 1.0 machine learning framework was used to build the algorithm, the memory was 8G, the main frequency was 1.6GHz, and the simulation experiment was carried out on the windows 10 operating system.
[0145] 3.1 Parameter settings
[0146] The effectiveness of the algorithm is verified by three sets of numerical experiments. The three test sets include 50 self-owned orders and 5, 10, and 20 cloud orders inserted respectively. The test set contains 10 machines and 10 types of workpieces, and each workpiece has 3 processes to be processed. Among them, the self-owned orders have arrived at time 0, the arrival of cloud orders follows the Poisson distribution, the workpiece processing time follows the uniform distribution of [1,10], the price of the workpiece follows the uniform distribution of [150,250], the operating cost per unit time of each machine follows the uniform distribution of [1,4], the fixed production cost of each workpiece follows the uniform distribution of [25,60], and the unit penalty factor of the workpiece delay is set to 5. The delivery date of each workpiece = the average processing time of the workpiece + the current arrival time of the workpiece + the uniform distribution [0,40]. The order delay target weight of the present invention is 0.8, the production disturbance target weight is 0.2, C1 = -100, C2 = -150.
[0147] Table 6 shows the experimental parameter settings of the order-taking agent. The activation function Relu has the advantages of low computational complexity and can solve the problem of gradient disappearance, so this paper selects Relu as the activation function. The selection of learning rate, ε random number, γ, Batch_size, optimizer, etc. has been debugged many times in the problem, and Table 6 and Table 7 are set as the final parameter tables.
[0148] Table 6 Experimental parameters of order-taking agent
[0149] Number of action spaces Learning Rate Activation Function ε random number γ Batch_size Optimizer 2 5e-5 Relu 0.1 0.99 32 Adam
[0150] Table 7 Experimental parameters of the order-arranging agent
[0151] Number of action spaces Learning Rate Activation Function ε random number γ Batch_size Optimizer 50 5e-5 Relu 0.05 0.99 32 Adam
[0152] 3.2 Comparison of algorithms and evaluation indicators
[0153] 3.2.1 Comparison of algorithms
[0154] In the experiment, the scheduling strategies obtained by the order-taking agent and the order-arranging agent algorithms are compared with the following methods:
[0155] SPT heuristic rule: Each process of each workpiece is completed by the machine with the shortest processing time, and the processing order of the workpiece is randomly selected. Without changing the production plan of the own orders, cloud orders are produced according to the earliest processing time of the machine.
[0156] Q-learning algorithm: The Q-learning algorithm based on linear approximation is used to represent the order-taking agent and the order-arranging agent, using the ε-greedy strategy. The characteristic function of the order-taking agent is the number of cloud orders × the number of actions, where the number of actions is 2, that is, accepting cloud orders and rejecting cloud orders. The state of the order-arranging agent remains unchanged, and the actions are a combination of the rules for selecting workpieces and the rules for selecting machines, with a total of 50.
[0157] DQN algorithm: The scheduling agent did not adopt strategy improvement, that is, it did not select the process candidate set and machine candidate set in advance. The scheduling agent selected the workpiece sequence as the processing sequence of the workpiece on the machine.
[0158] 3.2.2 Evaluation indicators
[0159] 1) Comparison of the best solutions: Each algorithm was iterated 200 times in three sets of numerical experiments, and the profit, machine load balance, and delay of all orders of the best solution were compared.
[0160] 2) Comparison of average solutions: Compare the average profit and average order acceptance rate of each algorithm.
[0161] 3) Worst solution comparison: Take out the worst profit after the algorithm runs and its total delay time.
[0162] 4) Comparison of different unit time delay costs: Change the penalty factor of the unit time delay cost and compare whether each algorithm can effectively adjust the order acceptance strategy and reduce the impact on the overall profit of the enterprise.
[0163] 3.3 Scheduling Strategy Experimental Results
[0164] When the number of self-owned orders is 50 and the number of cloud orders is 5, it follows the Poisson distribution of λ=0.5. When the number of cloud orders is 10, it follows the Poisson distribution of λ=1.3, and when the number is 20, it follows the Poisson distribution of λ=1. The three sets of experiments were run 200 times on each algorithm, and the results of various situations were statistically analyzed, and the results are as follows:
[0165] Table 8 Statistics of three groups of solutions
[0166]
[0167]
[0168] As can be seen from the above table, the best solution and average solution of the improved DQN algorithm are better than other algorithms. After the algorithm is improved, the profit of the solution and the overall order acceptance rate are greatly improved, which shows the effectiveness of the improvement strategy of the present invention. When the data set is small, the profit of the worst solution of the heuristic rule SPT is higher than that of the improved algorithm DQN, but when the data set gradually increases, the worst solution of the improved algorithm is better than SPT. When Q-learning and DQN do not have any improvement strategies, the best solution is not much different from the SPT rule, but the worst solution and average solution perform poorly, and the order acceptance rate is low.
[0169] Table 9 The best machine load
[0170]
[0171] The above table shows the cumulative processing time on each machine. It is easy to observe from the table that although the SPT rule has a short total processing time, the machine load is uneven, and the workload of machine 4 is much greater than that of machines 9 and 10. The Q-learning algorithm also has uneven machine loads. The short total processing time of this algorithm is mainly due to the small number of orders. The machine load of the algorithm DQN is more uniform, but the total processing time is large, and the order acceptance is unstable. Compared with the improved DQN algorithm before improvement, when the orders in numerical experiments 1 and 2 are the same, the total processing time of the solution obtained is greatly improved, the machine load is uniform, and the order acceptance of the best solution is stable. Compared with the SPT rule, the total processing time of the algorithm of the present invention is longer, but the profit of the best solution is much higher than that of the SPT rule, which means that when the algorithm is scheduled, machines with relatively low processing costs are selected, indicating the effectiveness of the algorithm of the present invention.
[0172] When there are 5 cloud orders, the Gantt chart obtained by the improved DQN algorithm, where the cloud orders are numbered from 50 to 54, Figure 3 Taking the upper left corner as an example, the first row J24 represents the workpiece numbered 24, and the second row 01 represents the first processing step.
[0173] Figure 4 In the Gantt chart obtained by the improved DQN algorithm, when there are 10 cloud orders, the cloud orders are numbered from 50 to 59.
[0174] Figure 5 In the Gantt chart obtained by the improved DQN algorithm, when there are 20 cloud orders, the cloud orders are numbered from 50 to 69.
[0175] like Figure 3-5 In the Gantt chart, the workpieces with short delivery time are processed by the machine first, and the workpieces with long delivery time are arranged to be processed later. When the workpiece is selected for machine processing, the machine with relatively low processing cost is selected, which increases the profit. The dynamically arriving cloud orders make full use of the remaining processing capacity of the workshop, and the algorithm can arrange production together with the own orders.
[0176] When an order is delayed, a penalty cost will be incurred, which will affect the profit. Therefore, the algorithm will generate different scheduling plans for different penalty factors. By changing the penalty factor of the delayed unit, the learning ability of each algorithm is demonstrated through the average profit and maximum profit of each algorithm.
[0177] like Figure 6-7 As shown in the figure, the improved DQN algorithm is always better than the other three algorithms in terms of average profit and maximum profit. As the penalty factor gradually increases, the average profit of the improved algorithm fluctuates slightly, but the maximum profit is very stable. The SPT algorithm shows a linear downward trend, while the average profits of the DQN and Q-learning algorithms are not only always lower than the SPT heuristic rule, but also decrease faster than the SPT algorithm. The maximum profit of the DQN algorithm is better than the heuristic rule SPT and Q-learning algorithm, but the performance is unstable. In summary, the improved DQN algorithm can continuously learn, and the order-taking agent and the order-scheduling agent can cooperate with each other, adjust the order acceptance strategy and order-scheduling strategy in real time, minimize the impact of the delay penalty cost on the profit, and ensure that the enterprise can obtain the maximum profit.
[0178] The present invention takes enterprises in the cloud platform as the background, and aims at the order acceptance and order scheduling problems of enterprises when the cloud platform dynamically dispatches orders. An improved DQN algorithm is proposed, and an order-accepting agent and an order-scheduling agent are designed. When the order-accepting agent makes a decision, the cloud order information and the workshop status information are integrated, and finally the order acceptance situation is fed back to the order-scheduling agent. The order-scheduling agent optimizes the strategy of the agent by combining the process candidate set, the machine candidate set, and arranging the workpiece to the earliest processing time of the selected machine. The improved DQN algorithm is compared with the heuristic rule SPT, the Q-learning algorithm, and the DQN algorithm. Numerical experiments show that the algorithm of the present invention is stable and superior to other algorithms in terms of maximum profit and average profit under different delay penalty factors, with a high order acceptance rate and balanced machine load. When the number of cloud orders increases, the worst solution of the algorithm of the present invention is also better than other algorithms.
[0179] The above illustrates the effectiveness of the present invention. The scheduling strategy of the intelligent agent can optimize the scheduling of dynamically arriving cloud orders, improve the utilization rate of workshop resources while producing its own orders normally, and increase the profit of the enterprise.
Claims
1. A cloud order dynamic acceptance and scheduling method based on deep reinforcement learning, characterized by Follow these steps: Step 1: The order-taking agent formulates an order acceptance strategy based on the information feedback from the workshop with the goal of maximizing profit; Step 2: When the cloud order is accepted, the order scheduling agent will reschedule the remaining processes and cloud orders at the current moment with the minimum delay time and minimum disturbance as the integration goal; after the scheduling is completed, the production capacity of the workshop will be fed back to the order receiving agent; In step 1, the objective function of profit is as follows: Among them, p b ,v,d b 、F b 、c b denote the price of the b-th cloud order, the penalty factor of the unit delay time, the delay time, the fixed production cost, and the machine production cost respectively; p Q d Q 、F Q 、c Q denote the price, delay time, fixed production cost, and machine production cost of the Q-th own order, respectively, where b = 1, 2, ... g, Q = 1, 2, ... h; In workshop scheduling, the minimum objective function for the cumulative delay time of all orders is: The objective function for minimizing workshop production disturbance is the cumulative difference in process time between the new and old scheduling schemes: In the formula, It represents the difference in the start processing time between the new and old scheduling plans for the wth process of the zth order. It represents the difference between the end processing time of the new scheduling plan and the old scheduling plan for the wth process of the zth order; where z=1,2,…L, w=1,2,…u; The Markov decision process G(S, A, P, R, γ) is used to describe the cloud order dynamic scheduling problem, which includes the state space S, action space A, state transition probability P, reward function R, and discount factor γ; In step 1, the reward function of the order-taking agent is The order-taking agent takes profit maximization as its objective function. Each order has a corresponding price, fixed cost, machine processing cost, and delay cost. The order-taking agent needs to determine whether there is profit in accepting cloud orders based on the status of the workshop. The reward function is designed as follows: In step 2, the reward function of the queue agent is After accepting an order, when the order scheduling agent inserts a new order into production, it will affect the production schedule of the existing orders. It not only needs to consider the order delay problem, but also the disturbance problem of the newly inserted order. Therefore, the order scheduling agent takes the dual objectives of minimum order delay and minimum order insertion disturbance as the objective function. In the above formula, the calculation method of the delay reward for each action is that when the workpiece is not completed, the delay time is calculated based on the latest start time of the process; when the workpiece is completed, the delay time is calculated based on the delivery date of the workpiece; where S is the cumulative total number of orders of the own orders and cloud orders, and r1 represents the order-taking agent at the decision time t k -1 transfer to t k The reward value; it is proved that minimizing the order delay time is equal to maximizing the reward function In the above formula, t j represents the delay time of the jth order, where j = 1, 2, … S; When inserting cloud orders into production, it will cause changes in machinery, equipment, personnel, and materials. Therefore, when making decisions, it is expected that the overall arrangement of the order will be as different from the previous production as possible. The overall difference is calculated by the difference between the start time and the end time of each process of each workpiece, and the time difference of the process production is fed back to the order scheduling agent as a reward; Convert the single-step reward into the overall objective function: The smaller the difference between the order production arrangement of the new scheduling plan and the old scheduling plan, the smaller the disturbance of the workshop will be. Minimizing the production disturbance of order insertion is equal to maximizing the reward function When a new order comes in and the order-scheduling agent makes a new scheduling arrangement, the final reward will weigh the integrated reward functions r1 and r2. The weighted method is used to integrate the two goals, and the reward function needs to be normalized first. The method is as follows: Since the maximum value of the delay reward function and the maximum value of the order disturbance function are unknown, they are set as: the delay reward function value r1 max =0, Order perturbation function value Among them, C1 and C2 are small negative numbers and are smaller than all reward function values; Convert the two objectives into a single objective optimization problem: In the above formula, w e represents the weight of the e-th target, r′ e Represents the normalized value of the e-th target value, where e=1,2.
2. The cloud order dynamic acceptance and scheduling method based on deep reinforcement learning as claimed in claim 1, characterized in that: In step 1, the state space of the order-taking agent is the processing end time O of each machine. i , Cumulative processing time of each machine Number of cloud order artifacts n b 、Workpiece type type(b)、Delivery date D b 、Price b ; In step 2, the state space of the order-scheduling agent is the processing end time O of each machine. i , Cumulative processing time of each machine The latest processing end time of each workpiece Remaining process E for each workpiece e , the processing schedule JM of each workpiece type on the machine; Among them, 1≤i≤m, 1≤e≤n, m is the number of machines, and n is the workpiece to be processed.
3. The cloud order dynamic acceptance and scheduling method based on deep reinforcement learning as claimed in claim 2 is characterized in that: In step 1, the scheduling actions of the order-taking agent are accepting orders and rejecting orders, which are defined as: In step 2, the order scheduling agent first selects the rules for processing workpieces and machines at the same time, then selects the workpiece to be processed from the process candidate set according to the rules, and then selects the processing machine from the machine candidate set.
Citation Information
Patent Citations
Multi-target flexible job shop scheduling method and device based on deep reinforcement learning
CN112149987A
Credit factory order scheduling method and device based on multi-agent reinforcement learning
CN112288341A
Multi-agent deep reinforcement learning scheduling method for textile fabric dyeing workshop
CN112633772A
MTO enterprise order acceptance method based on average reward reinforcement learning
CN113077188A
