Dynamic scheduling method for digital twin flexible job shop

Abstract

The invention provides a dynamic scheduling method for a digital twin flexible job shop, which comprises the following steps: acquiring physical shop data, and establishing a corresponding virtual shop model; extracting features of the physical workshop data to obtain state features of the physical workshop; performing disturbance prediction on the state characteristics of the physical workshop to obtain a disturbance prediction result; inputting the disturbance prediction result into a pre-trained dynamic scheduling model, and outputting an active scheduling result; inputting the active scheduling result and a passive scheduling result after disturbance into a virtual workshop model for simulation comparison to obtain an optimal scheduling strategy, feeding back the optimal scheduling strategy to the physical workshop, and executing the optimal scheduling strategy; wherein the training step of the dynamic scheduling model comprises the steps of training a machine agent and a workpiece agent by adopting a multi-agent PPO algorithm, converting a dynamic flexible job shop scheduling problem into a Markov decision solving process, and constructing a state space, a dynamic space and a reward function of the Markov decision process.

Description

Technical Field

[0001] The present invention relates to the technical field of production scheduling, and in particular to a dynamic scheduling method for a digital twin job shop. Background Art

[0002] In modern manufacturing, the flexible job shop (FJSP) scheduling problem is a typical complex combinatorial optimization problem that aims to optimize the utilization of workshop resources and improve production efficiency. However, traditional scheduling methods usually assume that the operating environment is static and deterministic, ignoring the dynamic uncertainties in actual production, such as machine failures, order changes, and worker errors. These disturbances can cause production plans to deviate from expectations, increase production costs, and extend construction periods. Therefore, how to deal with scheduling problems in a dynamic environment has become a challenge that needs to be urgently addressed in the field of production scheduling.

[0003] With the development of science and technology, flexible job shops have more complex and changeable scheduling states, which makes it more difficult to achieve effective flexible job shop scheduling. Therefore, how to build an efficient flexible job shop dynamic scheduling method has become a research hotspot. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a dynamic scheduling method for a digital twin flexible job shop, which can solve the scheduling deviation problem caused by machine failure in the dynamic flexible job shop scheduling problem (DFJSP).

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] On the one hand, the present invention provides a dynamic scheduling method for a digital twin flexible job shop, comprising:

[0007] Obtain physical workshop data and establish the corresponding virtual workshop model;

[0008] Extracting the characteristics of the physical workshop data to obtain the state characteristics of the physical workshop;

[0009] Performing disturbance prediction on the state characteristics of the physical workshop to obtain a disturbance prediction result;

[0010] Inputting the disturbance prediction result into a pre-trained dynamic scheduling model and outputting an active scheduling result;

[0011] The active scheduling result and the passive scheduling result after the disturbance occur are input into the virtual workshop model for simulation comparison to obtain the optimal scheduling strategy, which is fed back to the physical workshop to execute the optimal scheduling strategy;

[0012] The training steps of the dynamic scheduling model include:

[0013] The multi-agent PPO algorithm is used to train machine agents and workpiece agents, and the dynamic flexible job shop scheduling problem is transformed into solving a Markov decision process. The state space, dynamic space and reward function of the Markov decision process are constructed.

[0014] Optionally, the physical workshop updates data in real time and sends real-time status data to the virtual workshop model when executing the optimal scheduling strategy;

[0015] The virtual workshop model optimizes the scheduling strategy based on real-time status data and feeds back the optimal scheduling strategy to the physical workshop.

[0016] Optionally, the dynamic scheduling model is expressed as:

[0017] ;

[0018] The constraints are:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] In the formula, Indicates the maximum completion time of the workpiece; , Indicates the workpiece number; Representation of workpiece The corresponding process number; Representation of workpiece The corresponding process number; Indicates the machine number; Indicates the number of workpieces; Indicates the number of failures; Indicates The completion time of each workpiece; Indicates the process In the machine The processing time to complete the process; Indicates the process In the machine Processing time on Indicates the process A collection of machines to choose from; If the process In the machine 1 if the process is done, otherwise 0; Indicates the process In the machine Start processing time on Represents a preset positive integer; Representation of workpiece Arrival time; Indicates the process In the machine The processing time to complete the process; If the process In process Previously on the machine 1 if the process is done, 0 otherwise; Indicates a preset integer; Indicates the process A collection of machines to choose from; Indicates the process In the machine Start processing time on Indicates the process In the machine The processing time to complete the process; Representation of workpiece The completion time of each process; Indicates The time when the failure occurred; Indicates The time required to repair the fault; Representation of workpiece Delivery period; Representation of workpiece The delivery urgency factor; Representation of workpiece process.

[0033] Optionally, the multi-agent PPO algorithm is used to train the machine agent and the workpiece agent, including:

[0034] The workpiece agent selects a machine in the machine agent for processing according to the current processing state of the workpiece, and updates the machine buffer area;

[0035] An Actor strategy network is used to extract the probability distribution of workpiece actions under the current processing state of the workpiece and execute a scheduling strategy;

[0036] The critic value network is used to obtain the reward value of the current state of the workpiece and update the next processing state of the workpiece;

[0037] Sending the scheduling strategy, the current state reward value of the workpiece, the current processing state of the workpiece and the next processing state of the workpiece to the workpiece buffer area;

[0038] When the capacity of the workpiece buffer and the machine buffer reaches a preset threshold, the parameters of the Actor strategy network and the Critic value network are updated;

[0039] Repeat the process until all workpieces in the machine buffer are scheduled, and a trained machine agent and workpiece agent are obtained.

[0040] Optionally, the state space includes two machine state features and four workpiece state features, expressed as:

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] In the formula, express Time Machine Utilization rate; representing a first machine state characteristic; indicating a second machine state characteristic; Indicates a first workpiece state feature; Indicates a second workpiece state feature; express Time workpiece The completion rate of the number of processes; Indicates the third workpiece state feature; Indicates the fourth workpiece state characteristic; express Time workpiece The number of completed operations; express Time Machine The completion time of the last process; Indicates the average utilization of the machine; Indicates the number of machines; represents the standard deviation of machine utilization; It represents the average man-hour completion rate of all workpieces; It represents the standard deviation of the average labor completion rate of all workpieces; express Time workpiece The completion rate of working hours; It represents the average number of process completion rate of all workpieces; Represents the standard deviation of the completion rate of all workpiece processes; express Time workpiece process completion rate.

[0050] Optionally, the action space includes fourteen machine scheduling rules and ten workpiece scheduling rules.

[0051] Optionally, the reward function is expressed as:

[0052] ;

[0053] In the formula, represents the first weight coefficient; represents the second weight coefficient; represents the global reward; Indicates a partial reward.

[0054] Optionally, taking the optimal value of the maximum completion time of the workpiece as a reference, when the maximum completion time of the workpiece at the current moment is less than the optimal value of the maximum completion time of the workpiece, the difference between the optimal value of the maximum completion time of the workpiece and the maximum completion time of the workpiece at the current moment is used as the global reward value, and the optimal value of the maximum completion time of the workpiece is updated to the maximum completion time of the workpiece at the current moment; otherwise, the global reward value is zero.

[0055] Optionally, the average machine utilization is expressed as a function of the local reward:

[0056] ;

[0057] In the formula, represents the average utilization of the machine at the current decision moment; Represents the average machine utilization at the last decision moment.

[0058] In another aspect, the present invention provides a computer system comprising:

[0059] Memory, for storing computer instructions;

[0060] A processor is used to execute the computer instructions to implement the steps of the dynamic scheduling method of the digital twin flexible job shop described in the first aspect.

[0061] Beneficial Effects

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention solves the scheduling deviation problem caused by machine failure in the dynamic flexible job shop scheduling problem (DFJSP) by introducing digital twin technology and multi-agent PPO algorithm; by real-time monitoring of the dynamic data of the physical workshop and performing disturbance prediction, a virtual workshop model is constructed, which can quickly trigger the disturbance response mechanism when a machine failure occurs and generate an active scheduling plan. The present invention has a significant improvement effect in improving production efficiency and reducing scheduling plan deviations caused by machine failures, and provides a more efficient and intelligent scheduling solution for flexible job shops. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Shown is a schematic flow chart of a dynamic scheduling method for a digital twin flexible job shop of the present invention in one embodiment;

[0065] Figure 2 The present invention is shown in one embodiment when a machine fails while idle, and Schematic diagram of the prediction results when ;

[0066] Figure 3 The present invention is shown in one embodiment when a machine fails while idle, and Schematic diagram of the prediction results when ;

[0067] Figure 4 Shown is a schematic diagram of the prediction results of failures that occur during machining by the machine of the present invention;

[0068] Figure 5 FIG. 1 is a schematic diagram of a training process of a dynamic scheduling model of the present invention in one embodiment;

[0069] Figure 6It is a schematic diagram of structural parameters of a machine agent and a workpiece agent in one embodiment of the present invention;

[0070] Figure 7 It is a schematic diagram of the iterative process of the machine agent and the workpiece agent in one embodiment of the present invention;

[0071] Figure 8 FIG. 1 is a schematic diagram of an initial scheduling scheme based on a multi-intelligent PPO algorithm in an embodiment of the present invention;

[0072] Fig. 9 It is a schematic diagram of an active scheduling scheme based on digital twin technology and multi-intelligent PPO algorithm in one embodiment of the present invention;

[0073] Fig.10 It is a schematic diagram of a passive rescheduling scheme of the initial scheduling based on the multi-intelligent PPO algorithm after the disturbance occurs in one embodiment of the present invention;

[0074] Fig.11 Shown is a schematic diagram of a global rescheduling solution based on digital twin technology and multi-intelligent PPO algorithm after a disturbance occurs in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0076] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0077] Example 1

[0078] like Figure 1 As shown, this embodiment introduces a dynamic scheduling method for a digital twin flexible job shop.

[0079] The dynamic scheduling process of traditional workshop systems is usually "generate initial scheduling plan - monitor production status - plan adjustment - execute new plan". For random disturbances in the workshop, the response is to readjust or calculate the scheduling plan based on workshop information after the disturbance occurs. This passive response often leads to decision delays, which affects production efficiency. At the same time, this dynamic scheduling method has deficiencies in predictability, initiative and visualization, and it is difficult to meet the needs of smart manufacturing workshops for flexibility and efficiency.

[0080] This embodiment applies digital twin technology to the dynamic scheduling problem of the workshop and proposes a dynamic scheduling framework, such as Figure 1 Through the workshop dynamic scheduling framework, the workshop scheduling problem is transformed into a Markov decision process, and combined with digital twin technology, early perception and active response to disturbances are achieved, which specifically includes the following steps:

[0081] Through various sensors or communication protocols, data is collected from the physical workshop to obtain physical workshop data, so as to achieve real-time perception of various production factors in the workshop and build a corresponding high-fidelity virtual workshop model based on the actual physical workshop situation.

[0082] The original data of the workshop is preprocessed and fused to better reflect the real status of the workshop; on this basis, feature extraction is performed to obtain the status characteristics of the workshop.

[0083] The disturbance prediction is performed using the state characteristics of the physical workshop to obtain the disturbance prediction results.

[0084] The disturbance prediction result is input into a pre-trained dynamic scheduling model, and an active scheduling result is output.

[0085] The active scheduling result and the passive scheduling result after the disturbance occur are input into the virtual workshop model for simulation comparison to obtain the optimal scheduling strategy, which is fed back to the physical workshop through the virtual-real information interaction mechanism to execute the optimal scheduling strategy.

[0086] Among them, active scheduling refers to the rescheduling of workshop resources before the disturbance occurs through disturbance prediction, so as to achieve active avoidance of potential disturbances; passive scheduling refers to responding to the disturbance and continuing to execute the initial scheduling plan after the disturbance occurs, which is generally manifested as a simple right shift and rescheduling.

[0087] When executing the optimal scheduling strategy, the physical workshop updates data in real time and sends real-time status data to the virtual workshop model; the virtual workshop model optimizes the scheduling strategy according to the real-time status data, and feeds back the optimal scheduling strategy to the physical workshop.

[0088] That is, in this embodiment, the physical workshop continuously updates data during the scheduling execution process and transmits the latest status to the virtual workshop. The virtual workshop continuously optimizes the decision plan based on the real-time status and feeds back the optimal decision to the physical workshop. This process forms a complete closed-loop information interaction mechanism, ensuring the continuous iteration and synchronous update of information between the virtual and physical workshops. By implementing this efficient virtual-real interaction process, dynamic optimization and continuous improvement of the virtual and physical workshops are achieved.

[0089] The disturbance type considered in dynamic scheduling in this embodiment is mainly machine failure. The following processing is performed for the predicted machine failure:

[0090] Depending on the state of the machine when the fault is predicted, it is necessary to determine whether the fault will affect the normal production of the machine. Assume that the time required to repair the machine after the fault is T, the predicted time of the fault is B, the idle time of the machine is L, and the time to complete the previous process on the machine is .

[0091] If the machine is predicted to fail while idle, it will go directly to the maintenance phase. , it does not affect the production plan of the machine, and the workshop still produces according to the initial plan. Figure 2 As shown; if , then the initial production schedule is affected, and the workpieces to be processed in the buffer area need to select other machines that can be processed, such as Figure 3 shown.

[0092] If a machine is predicted to fail during processing, the machine will be repaired immediately, the workpiece being processed needs to exit the current process and be reprocessed on a machine that can be processed. The workpieces in the original buffer area need to be re-selected for other machines that can be processed and the scheduling plan needs to be re-made. During rescheduling, the machine failure is uniformly regarded as a process with no substantial tasks of duration T and assigned to the failed machine, such as Figure 4 shown.

[0093] The dynamic flexible job shop scheduling problem (DFJSP) considering machine failure is described as follows: During the processing, Workpieces exist Machine On processing, each workpiece There are many different processes recorded as Each process must be processed in strict sequence. Each workpiece contains at least one process, and each process has multiple machines to choose from. In order to reduce the impact of disturbances and improve production efficiency, the maximum completion time of the workpiece is In order to optimize the goal, DFJSP also needs to meet the following constraints:

[0094] (1) At the beginning, all machines and workpieces are ready;

[0095] (2) The same machine can only process the same workpiece at the same time;

[0096] (3) The same process of the same workpiece can only be processed by the same machine at the same time;

[0097] (4) There are disturbances caused by machine failures during the processing;

[0098] (5) Different artifacts have the same priority and no order constraints;

[0099] (6) Each machine has the same probability of failure;

[0100] (7) After a machine fails, the recovery time can be predicted and the machine can operate normally after recovery.

[0101] (8) When a machine fails, the workpiece being processed on the failed machine needs to be withdrawn and then assigned to another available machine for reprocessing. The machine can only be selected after the repair is completed.

[0102] In order to solve the DFJSP, it is necessary to abstract the problem through mathematical modeling. The dynamic scheduling model is expressed as:

[0103] ;

[0104] The constraints are:

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] In the formula, Indicates the maximum completion time of the workpiece; , Indicates the workpiece number; Representation of workpiece The corresponding process number; Representation of workpiece The corresponding process number; Indicates the machine number; Indicates the number of workpieces; Indicates the number of failures; Indicates The completion time of each workpiece; Indicates the process In the machine The processing time to complete the process; Indicates the process In the machine Processing time on Indicates the process A collection of machines to choose from; If the process In the machine 1 if the process is done, 0 otherwise; Indicates the process In the machine Start processing time on represents a preset positive integer, which is a very large positive integer in this embodiment; Representation of workpiece Arrival time; Indicates the process In the machine The processing time to complete the process; If the process In process Previously on the machine 1 if the process is done, 0 otherwise; represents a preset integer, which is a sufficiently large integer in this embodiment; Indicates the process A collection of machines to choose from; Indicates the process In the machine Start processing time on Indicates the process In the machine The processing time to complete the process; Representation of workpiece The completion time of each process; Indicates The time when the failure occurred; Indicates The time required to repair the fault; Representation of workpiece Delivery period; Representation of workpiece The delivery urgency factor; Representation of workpiece process.

[0119] The constraints indicate that the processing time of any process is non-negative and the process It can only be assigned to one machine. When the machine has no processing tasks, the start and end times are both 0. The difference between the completion time and the start time is greater than the processing time, the workpiece starts processing after it arrives, and the process and process The same workpiece will not be processed on the same machine at the same time, the same workpiece will be processed in sequence, the completion time of the workpiece, the machine cannot be processed when it is in a faulty or repaired state, and the workpiece delivery time.

[0120] The training steps of the dynamic scheduling model include:

[0121] The multi-agent PPO algorithm is used to train machine agents and workpiece agents, transform the dynamic flexible job shop scheduling problem into a Markov decision process, and construct the state space, dynamic space and reward function of the Markov decision process. The idea is as follows:

[0122] First, the workpiece agent obtains the current processing status from the workshop environment , select the machine in the machine agent for processing and update the machine cache; then the policy network outputs the probability distribution of each action taken in the current state to guide the agent to perform appropriate actions; finally, the value network is used to feedback a reward value to the agent And update the next state ; In the continuous interaction process, data such as state, action, reward and probability distribution of action will be stored in the artifact cache area. The policy network and value network are updated according to the data in the experience playback cache area. When the capacity of the artifact buffer and the machine buffer reaches the preset threshold, the parameters of the policy network and the value network are updated; repeat the execution until all artifacts in the machine cache area are scheduled, and the trained machine agent and artifact agent are obtained. The policy network can be an Actor policy network, and the value network can be a Critic value network.

[0123] The state space of workshop scheduling usually describes the current processing status and resource allocation of the workshop, so that the agent can make decisions based on this information. In order to better observe the status of workpieces and machines and make more reasonable decisions, this embodiment designs machine agents and workpiece agents respectively. The machine agent selects the next workpiece to be processed for the machine that has completed the processing, and the workpiece agent selects an available machine for the workpiece according to the current processing status. Based on the working principles and objective functions of the two agents, this embodiment designs six state characteristics, including two machine state characteristics and four workpiece state characteristics, which more comprehensively reflect the processing status of the machine and the completion status of the workpiece. All state feature values ​​need to be normalized to improve the stability and versatility of the model. The two machine state characteristics and four workpiece state characteristics are expressed as:

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] In the formula, express Time Machine Utilization rate; representing a first machine state characteristic; indicating a second machine state characteristic; Indicates a first workpiece state feature; Indicates a second workpiece state feature; express Time workpiece The completion rate of the number of processes; Indicates the third workpiece state feature; Indicates the fourth workpiece state characteristic; express Time workpiece The number of completed operations; express Time Machine The completion time of the last process; Indicates the average utilization of the machine; Indicates the number of machines; represents the standard deviation of machine utilization; It represents the average man-hour completion rate of all workpieces; It represents the standard deviation of the average labor completion rate of all workpieces; express Time workpiece The completion rate of working hours; It represents the average number of process completion rate of all workpieces; Represents the standard deviation of the completion rate of all workpiece processes; express Time workpiece process completion rate.

[0133] The action space is the set of all possible actions that the agent can choose. In the workshop scheduling problem, the action space is usually defined according to a specific scheduling algorithm or rule. Therefore, this embodiment is based on the scheduling rule objective function, and designs reasonable scheduling rules for machine agents and workpiece agents, respectively, as shown in Tables 1 and 2. In this embodiment, the action space includes fourteen machine scheduling rules and ten workpiece scheduling rules to determine the optimal workpiece allocation plan and production scheduling strategy.

[0134] Table 1 Artifact rules of machine agents

[0135]

[0136] Table 2 Machine rules of workpiece agent

[0137]

[0138] The reward function is related to the objective function, reflecting the quality of the agent's behavior and guiding the agent to train towards the task goal. When solving traditional workshop scheduling problems, the agent is usually rewarded only when it successfully completes the entire task. However, when dealing with DFJSP with the maximum completion time as the optimization goal, the use of traditional reward mechanisms may require a longer training cycle, resulting in the sparsity of rewards. Therefore, local rewards are also considered on the basis of global rewards. Local rewards are to obtain instant feedback when the agent completes part of the task or local goals, which effectively alleviates the problem of reward sparsity; global rewards guide the overall training of the agent towards the goal and improve the convergence speed. Therefore, the reward function is expressed as:

[0139] ;

[0140] In the formula, represents the first weight coefficient; represents the second weight coefficient; represents the global reward; represents a partial reward; and .

[0141] In order to guide the agent to train towards the global optimal goal, a value negatively correlated with the maximum completion time is used as the global reward, with the optimal value of the maximum completion time As a benchmark, (if there is no optimal value, the result of the optimal single rule is needed as the optimal solution), the maximum completion time obtained at the current moment is ,if , then the difference between the two is fed back As the reward value, the optimal completion time Updated to ; Otherwise, set the global reward to 0 directly.

[0142] Since the tasks of the machine agent and the workpiece agent are different, it is necessary to design local reward functions related to the completion time for them respectively. The average utilization rate of the machine can reflect the maximum completion time, which is generally negatively correlated. Therefore, the average utilization rate of the machine is selected as the local reward function for the machine agent. Compared with the previous decision moment, if the average utilization rate of the machine at the current decision moment increases, the machine agent is given a positive reward; if it decreases, it is punished; otherwise, there is no reward, which is expressed as:

[0143] ;

[0144] In the formula, represents the average utilization of the machine at the current decision moment; Represents the average machine utilization at the last decision moment.

[0145] For the workpiece agent, it is mainly responsible for selecting a processable machine for the workpiece. The final processing order is still determined by the machine agent. In essence, it is the machine agent that determines the maximum completion time. Therefore, in order to facilitate the training of DFJSP with the maximum completion time as the goal, the local reward of the workpiece agent is set to 0.

[0146] According to the different roles of machine agents and workpiece agents, the dynamic scheduling model and its training steps designed in this embodiment are as follows: Figure 5 shown.

[0147] In actual scheduling, when a new workpiece arrives, the workpiece agent selects an available processing machine for it according to the current processing status. In this embodiment, the workpiece agent is the workpiece Machine M3 is selected, and the workpiece buffer of M3 becomes Then for each machine, if there is a workpiece to be assigned in the machine buffer, the machine agent will calculate the workpiece according to its current processing status. , output the scheduling rules, select the workpiece to be processed in the buffer area, and obtain the reward value for this time , the workpiece enters the next state Repeat this process until all the workpieces in the buffer area have been scheduled. After machine M3 completes the processing of a workpiece, it will be scheduled according to the scheduling rules. Selected Workpiece Processing is performed, and the workpiece in the cache is updated to In the above process, each scheduling generates , , , Stored in the workpiece cache area, after the cache area capacity reaches the preset capacity, the machine agent and the workpiece agent use the PPO algorithm to update the parameters of the strategy network and the value network.

[0148] The network structure parameters of the two agents are as follows: Figure 6 As shown, the state features are input into the machine agent, and the rules for selecting workpieces are output, with a structural parameter of 6×64×32×14; the state features are input into the workpiece agent, and the rules for selecting machines are output, with a structural parameter of 6×64×32×10. This embodiment selects rules and updates parameters for the machine agent and the workpiece agent based on the multi-agent PPO algorithm, which is mainly divided into agent sampling, workshop environment update, reward function calculation, and network parameter update. After the training is completed, the algorithm generates the parameters of the two action networks without repeated training.

[0149] In order to train the proposed dynamic scheduling model, a randomly generated 5×10×5 case is simulated. The case parameters are shown in Table 3, where the processing time of each process on the machine follows the uniform distribution [1,20]. The algorithm parameter settings during training are shown in Table 4.

[0150] Table 3 Random case parameters

[0151]

[0152] Table 4 Algorithm parameter design

[0153]

[0154] like Figure 7As shown in the figure, the loss reduction process of the workpiece agent and the machine agent is shown. It can be seen that the loss values ​​of the two agents dropped rapidly in the first 1000 trainings. In the subsequent convergence process, there was a certain loss fluctuation. This is mainly because for combinatorial optimization problems such as DFJSP, each decision error of the agent in each round of scheduling will indirectly affect the status and decision choices of all subsequent scheduling points, thereby causing a sudden change in the global reward and causing the model's loss to fluctuate. Nevertheless, from the overall trend, the two agents eventually tended to converge.

[0155] In order to verify the superiority of the proposed algorithm, the parameters obtained by training are applied to the Brandimarte standard example for solution, and the running results are compared with the scheduling rule combination and meta-heuristic algorithm, as shown in Table 5. The scheduling rule combination includes SPT+FIFO, SRPT+FIFO, SRPT+SPT and a group of randomly generated scheduling rule combinations R+R, and the meta-heuristic algorithm selects the GA algorithm.

[0156] The results in Table 5 show that compared with the scheduling rule combination, the multi-agent PPO algorithm proposed in the present invention is significantly better than the scheduling rule combination in the MK01~MK10 examples; compared with the GA algorithm, it is slightly worse than the GA algorithm in the MK01 and MK03~MK06 examples, with an average difference of 5.35%. However, the result on MK02 is the same as the GA algorithm, and the result on the MK06~MK10 examples is slightly better than the GA algorithm, with an average difference of 6.59%. Therefore, the algorithm of this embodiment has obvious advantages over the scheduling rules and GA algorithm in solving DFJSP, especially when the complexity of the example increases, its advantages are more significant.

[0157] Table 5 Comparison of results

[0158]

[0159] The solution time of the algorithm in this paper is compared with that of the above algorithms in the classic example, and the results are shown in Table 6. Compared with the solution time of the metaheuristic algorithm in the classic example, although the algorithm of the present invention adds the process of building the workshop environment, its solution efficiency is improved by 30 times compared with the GA algorithm. Compared with the scheduling rule combination strategy, its decision time increases by 20 times, mainly due to the time spent on the forward propagation of the neural network and the update of the workshop environment.

[0160] Table 6 Comparison of solution time for classic use cases

[0161]

[0162] Based on the above analysis, although the multi-agent PPO algorithm and the GA algorithm have their own advantages and disadvantages in terms of solution effect, its solution speed is much better than GA, especially when dealing with complex use cases, the performance advantage of the algorithm is more obvious, which can effectively cope with workshop production tasks with high real-time requirements and complex environments.

[0163] Example 2

[0164] This embodiment introduces a specific experimental design of a dynamic scheduling method for a digital twin job shop, including:

[0165] The dynamic scheduling method proposed in this embodiment is applied and verified in a certain automobile controller machining workshop. The workshop mainly includes processing workpieces such as controller housing, upper cover, main control board bracket, copper plate, and processing machines such as CNC milling machines, CNC lathes, and CNC punching machines, involving stamping, milling, cutting and other processes.

[0166] The workshop scheduling system first solves the following problem based on the multi-agent PPO algorithm: Figure 8 The initial scheduling plan shown in the figure finally obtains a maximum completion time of 51 minutes.

[0167] In the digital twin workshop, the disturbance prediction method predicts that the data of machines M6 and M4 will be abnormal at 10 minutes and 20 minutes respectively. At this time, machines M4 and M6 are under maintenance and cannot be selected until the maintenance is completed. The estimated maintenance time of the machines is 12 minutes and 20 minutes respectively. and process It is necessary to exit first and then allocate other available machines for reprocessing. The remaining processes on the faulty machines and non-faulty machines form a new scheduling set. The reinforcement learning algorithm is used to generate an active scheduling plan. The final scheduling plan result is as follows Fig. 9 In the active scheduling scheme, by adjusting some processing steps, all workpieces are ensured to be processed on time, with the maximum completion time being 57 minutes, and the deviation rate from the initial scheme is only 11.76%.

[0168] The passive rescheduling plan of the initial scheduling plan under the actual failure is as follows: the actual occurrence time of the machine M6 failure is 12 minutes, the actual occurrence time of the machine M4 failure is 22 minutes, and the process and process It is necessary to exit first and wait until the repair of machines M4 and M6 is completed before reprocessing. The processing machines and process schedules will not be changed. Therefore, right shift and rescheduling are selected as the scheduling result after the failure. For example, Fig.10As shown in the figure. The maximum completion time of right-shift rescheduling is 78 minutes, and the completion period is extended by 52.94%, of which workpiece N6 is delayed by 10 minutes. Therefore, the active scheduling scheme is selected based on the dynamic scheduling results of digital twins. The fault data is input into the dynamic scheduling model based on digital twins, and the output scheduling scheme is consistent with the above scheme, which verifies the effectiveness of the dynamic scheduling model based on digital twins.

[0169] To further verify the superiority of the dynamic scheduling mechanism based on digital twins, the scheduling scheme of the present invention is also compared with the global rescheduling scheme after the disturbance occurs. and process It is necessary to quit and reprocess at 12 minutes and 22 minutes respectively. The reinforcement learning algorithm is also used to generate a rescheduling plan. The results are as follows Fig.11 As shown, the maximum completion time is 58 minutes, and the completion period is extended by 13.72%.

[0170] The dynamic scheduling of the workshop should not only consider the basic performance index of the maximum completion time of the rescheduling scheme, but also the stability of the scheduling scheme after the disturbance event occurs. Therefore, the three indicators of maximum completion time, time deviation and machine deviation are selected to evaluate the rescheduling scheme.

[0171] Timing Deviation (TD): reflects the change in the start time of each process before and after rescheduling. The smaller the time deviation, the better the stability of the rescheduling plan, expressed as:

[0172] ;

[0173] ;

[0174] Machine Deviation (MD): reflects the changes in the processing machines of each process before and after rescheduling. The smaller the machine deviation, the better the stability of the scheduling plan, expressed as:

[0175] ;

[0176] ;

[0177] Comprehensive evaluation indicators (CEI): It is a comprehensive evaluation of the maximum completion time, time deviation and machine deviation. The smaller the comprehensive evaluation indicator, the better the stability of the scheduling plan, which is expressed as:

[0178] ;

[0179] in, , , is the weight coefficient, and .

[0180] Calculate the evaluation indicators of the above three rescheduling schemes. Since dynamic scheduling focuses on the stability based on completion time, , , Taking 0.6, 0.2, and 0.2 respectively, the evaluation indicators are shown in Table 7.

[0181] Table 7 Evaluation indicators of three scheduling schemes

[0182]

[0183] As can be seen from Table 7, compared with the right-shift rescheduling after the disturbance, the active scheduling based on digital twins has increased time deviation and machine deviation, but the maximum completion time is reduced by 26.92%, and the comprehensive evaluation index is reduced by 20.66%. Therefore, the stability based on the maximum completion time is better. Compared with the complete rescheduling after the disturbance, the active scheduling reduces the maximum completion time by 1.7% when the time deviation and machine deviation are the same. This is mainly because it can detect the disturbance earlier and make adjustments, and make more reasonable arrangements for the workpiece process. It is effective in reducing the scheduling plan deviation caused by machine failure, which proves that this method effectively improves the stability of the workshop.

[0184] Example 3

[0185] This embodiment introduces a computer system, including:

[0186] Memory, for storing computer instructions;

[0187] A processor is used to execute the computer instructions to implement the steps of the dynamic scheduling method of the digital twin flexible job shop described in Example 1 or 2.

[0188] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0189] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0192] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A dynamic scheduling method for a digital twin flexible job shop, characterized in that: include: Obtain physical workshop data and establish the corresponding virtual workshop model; Extracting the characteristics of the physical workshop data to obtain the state characteristics of the physical workshop; Performing disturbance prediction on the state characteristics of the physical workshop to obtain a disturbance prediction result; Inputting the disturbance prediction result into a pre-trained dynamic scheduling model and outputting an active scheduling result; The active scheduling result and the passive scheduling result after the disturbance occur are input into the virtual workshop model for simulation comparison to obtain the optimal scheduling strategy, which is fed back to the physical workshop to execute the optimal scheduling strategy; The training steps of the dynamic scheduling model include: The multi-agent PPO algorithm is used to train machine agents and workpiece agents, and the dynamic flexible job shop scheduling problem is transformed into solving a Markov decision process. The state space, dynamic space and reward function of the Markov decision process are constructed.

2. The dynamic scheduling method of the digital twin flexible job shop according to claim 1 is characterized in that: When the physical workshop executes the optimal scheduling strategy, the physical workshop updates data in real time and sends real-time status data to the virtual workshop model; The virtual workshop model optimizes the scheduling strategy based on real-time status data and feeds back the optimal scheduling strategy to the physical workshop.

3. The dynamic scheduling method of the digital twin flexible job shop according to claim 1 is characterized in that: The dynamic scheduling model is expressed as: ； The constraints are: ； ； ； ； ； ； ； ； ； ； ； ； ； In the formula, Indicates the maximum completion time of the workpiece; , Indicates the workpiece number; Representation of workpiece The corresponding process number; Representation of workpiece The corresponding process number; Indicates the machine number; Indicates the number of workpieces; Indicates the number of failures; Indicates The completion time of each workpiece; Indicates the process In the machine The processing time to complete the process; Indicates the process In the machine Processing time on Indicates the process A collection of machines to choose from; If the process In the machine 1 if the process is done, 0 otherwise; Indicates the process In the machine Start processing time on Represents a preset positive integer; Representation of workpiece Arrival time; Indicates the process In the machine The processing time to complete the process; If the process In process Previously on the machine 1 if the process is done, otherwise 0; Indicates a preset integer; Indicates the process A collection of machines to choose from; Indicates the process In the machine Start processing time on Indicates the process In the machine The processing time to complete the process; Representation of workpiece The completion time of each process; Indicates The time when the failure occurred; Indicates The time required to repair the fault; Representation of workpiece Delivery period; Representation of workpiece The delivery urgency factor; Representation of workpiece process.

4. The dynamic scheduling method of the digital twin workshop according to claim 1 is characterized in that: The multi-agent PPO algorithm is used to train the machine agent and the workpiece agent, including: The workpiece agent selects a machine in the machine agent for processing according to the current processing state of the workpiece, and updates the machine buffer area; An Actor strategy network is used to extract the probability distribution of workpiece actions under the current processing state of the workpiece and execute a scheduling strategy; The critic value network is used to obtain the reward value of the current state of the workpiece and update the next processing state of the workpiece; Sending the scheduling strategy, the current state reward value of the workpiece, the current processing state of the workpiece and the next processing state of the workpiece to the workpiece buffer area; When the capacity of the workpiece buffer and the machine buffer reaches a preset threshold, the parameters of the Actor strategy network and the Critic value network are updated; Repeat the process until all workpieces in the machine buffer are scheduled, and a trained machine agent and workpiece agent are obtained.

5. The dynamic scheduling method of the digital twin flexible job shop according to claim 3 is characterized in that: The state space includes two machine state features and four workpiece state features, which are expressed as: ； ； ； ； ； ； ； ； In the formula, express Time Machine Utilization rate; representing a first machine state characteristic; indicating a second machine state characteristic; Indicates a first workpiece state feature; Indicates a second workpiece state feature; express Time workpiece The completion rate of the number of processes; Indicates the third workpiece state feature; Indicates the fourth workpiece state characteristic; express Time workpiece The number of completed operations; express Time Machine The completion time of the last process; Indicates the average utilization of the machine; Indicates the number of machines; represents the standard deviation of machine utilization; It represents the average man-hour completion rate of all workpieces; It represents the standard deviation of the average labor time completion rate of all workpieces; express Time workpiece The completion rate of working hours; It represents the average number of process completion rate of all workpieces; Represents the standard deviation of the completion rate of all workpiece processes; express Time workpiece process completion rate.

6. The dynamic scheduling method of the digital twin flexible job shop according to claim 1 is characterized in that: The action space includes fourteen machine scheduling rules and ten workpiece scheduling rules.

7. The dynamic scheduling method of the digital twin flexible workshop according to claim 1 is characterized in that: The reward function is expressed as: ； In the formula, represents the first weight coefficient; represents the second weight coefficient; represents the global reward; Indicates a partial reward.

8. The dynamic scheduling method of the digital twin flexible job shop according to claim 7 is characterized in that: Taking the optimal value of the maximum completion time of the workpiece as the benchmark, when the maximum completion time of the workpiece at the current moment is less than the optimal value of the maximum completion time of the workpiece, the difference between the optimal value of the maximum completion time of the workpiece and the maximum completion time of the workpiece at the current moment is used as the global reward value, and the optimal value of the maximum completion time of the workpiece is updated to the maximum completion time of the workpiece at the current moment; otherwise, the global reward value is zero.

9. The dynamic scheduling method of the digital twin flexible workshop according to claim 7 is characterized in that: The average machine utilization as a function of local reward is expressed as: ； In the formula, represents the average utilization of the machine at the current decision moment; Represents the average machine utilization at the last decision moment.

10. A computer system, characterized in that: include: Memory, for storing computer instructions; A processor, used to execute the computer instructions to implement the steps of the dynamic scheduling method of the digital twin flexible job shop described in any one of claims 1-9.

Images