A flexible job shop scheduling method, system and electronic device

Abstract

The application provides a flexible workshop scheduling method and system and electronic equipment, and relates to the technical field of discrete manufacturing intelligent scheduling.The application comprises the following steps: generating a plurality of flexible workshop environments in a production workshop according to a preset production target; constructing a scheduling strategy model of the production workshop based on a Markov decision process; simultaneously optimizing a feature extraction network, an Actor network and a Critic network by using the scheduling strategy model and a plurality of data sets; determining the scheduling plan corresponding to the maximum completion time as the optimal scheduling plan after the optimization is completed; and completing the preset production target based on the optimal scheduling plan.The application extracts features from a plurality of workshop environments respectively to generate a scheduling scheme, thereby improving the efficiency and rationality of flexible workshop scheduling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of discrete manufacturing intelligent scheduling, in particular to a flexible job shop scheduling method and system and an electronic device. BACKGROUND

[0002] Manufacturing intelligence is the main direction of innovation-driven and transformational upgrading of China's manufacturing industry, and the intelligent production scheduling is the key path to realize manufacturing intelligence. In manufacturing industry, the production capacity of an enterprise is closely related to the resource scheduling strategy adopted. Today, the market competition is becoming increasingly fierce, and the customer demand is complex and changeable, so a scheduling system with real-time, versatility, flexibility and scalability is needed to arrange production tasks to realize efficient use of production resources and maximize production efficiency. Therefore, it is of great theoretical significance and economic value to study the intelligent optimization scheduling and autonomous decision-making method of discrete manufacturing. The flexible job shop scheduling problem (FJSP) is a general scheduling problem as a generalization of the job shop scheduling problem (JSP), and it has attracted much attention in the industry because it meets the requirements of production flexibility and diversity in actual production scenarios.

[0003] The traditional methods for solving production scheduling problems mainly include exact methods, meta-heuristic methods and heuristic methods. These methods have certain application bottlenecks. For example, the exact methods such as branch and bound method and mathematical programming method can obtain the optimal solution of the original problem, but usually have exponential computational complexity, which cannot meet the real-time requirements of scheduling in actual production scenarios; the meta-heuristic methods such as genetic algorithm and particle swarm algorithm are widely used, but their performance is sensitive to parameters and has poor generalization; the heuristic method refers to a method that sets rules in advance based on prior knowledge to solve the original problem, which is simple to program, has good real-time performance and generalization, but the quality of the solution generated by this method is often not good enough and can only adapt to some specific scenarios.

[0004] In recent years, deep reinforcement learning methods have shown advantages in many fields, including combinatorial optimization fields such as scheduling problems. Reinforcement learning methods model the scheduling task as a Markov Decision Process (MDP), support agents to explore and learn in a simulated workshop environment, and are a data-driven method that can be implemented offline. At the same time, when the agent applies the learned strategy in the actual environment, it can quickly give the evaluation quantity with very small time cost. Therefore, this method has the advantages of data learning and real-time decision-making, effectively overcoming the shortcomings of traditional methods. At the same time, in order to make the decision model have the generalization ability to solve scheduling problems of different scales, scholars have applied different state representation methods in the design of the model, which has been successfully applied to the optimization of minimum completion time of flexible job shop scheduling problems. Han et al. designed a scheduling decision-making method based on pointer network and policy gradient algorithm by using an improved pointer network to encode and decode the information of the process to be scheduled. Lei et al. designed a feature extraction method based on isomorphic graph, learned the flexible workshop environment based on the disjunctive graph representation, divided the decision into process selection and machine selection, and set up two agents to handle the two-step decision-making. Song et al. proposed a heterogeneous disjunctive graph method to describe the flexible workshop environment, and designed an end-to-end scheduling strategy model based on heterogeneous graph neural network and proximal policy optimization algorithm, which is superior to simple scheduling rules and meta-heuristic algorithms in solution quality. However, the above-mentioned scheduling methods can be used to solve flexible job shop scheduling problems, but there are still some problems, mainly in the aspects of insufficient learning of production unit characteristics and insufficient exploration of flexible workshop environment, and there is room for improvement in solution quality, computational efficiency and generalization ability. SUMMARY

[0005] The purpose of the present application is to provide a flexible workshop scheduling method, system and electronic equipment, which can extract features from multiple workshop environments respectively to generate scheduling schemes, thereby improving the efficiency and rationality of flexible workshop scheduling.

[0006] To achieve the above-mentioned purpose, the present application provides the following scheme:

[0007] A flexible workshop scheduling method, comprising:

[0008] Randomly generating a plurality of flexible workshop environments in a production workshop according to a preset production target; constructing a plurality of data sets according to the parameters of the plurality of flexible workshop environments; the data set corresponds one-to-one to the flexible workshop environment; the data set is a training set or a validation set;

[0009] Constructing a scheduling strategy model of the production workshop based on a Markov Decision Process;

[0010] The feature extraction network, the Actor network and the Critic network are all subjected to parameter initialization processing;

[0011] The feature extraction network, the Actor network and the Critic network are simultaneously optimized by using the scheduling strategy model and the plurality of data sets, and after the optimization is completed, a scheduling plan corresponding to a maximum completion time is determined as an optimal scheduling plan;

[0012] The preset production target is completed based on the optimal scheduling plan.

[0013] The feature extraction network, the Actor network and the Critic network are simultaneously optimized by using the scheduling strategy model and the plurality of data sets, and after the optimization is completed, a scheduling plan corresponding to a maximum completion time is determined as an optimal scheduling plan, which comprises:

[0014] The initialized feature extraction network parameters, Actor network parameters and Critic network parameters are determined as network parameters of the 0th training round;

[0015] The evaluation quantity of the 0th training round is initialized;

[0016] Let the training round number Episode = 1;

[0017] The cache pool and the capacity of the scheduling strategy model are initialized;

[0018] Let the first iteration number i = 1;

[0019] Any training flexible workshop environment in the training flexible workshop environment set is determined as a current training flexible workshop environment; the training flexible workshop environment is a flexible workshop environment corresponding to the training set;

[0020] The process machining graph and the machine competition graph of the current training flexible workshop environment are determined;

[0021] According to the process machining graph and the machine competition graph, the cache pool is updated by using the feature extraction network, the Actor network and the Critic network;

[0022] The parameters of the feature extraction network, the Actor network and the Critic network are all updated by using the gradient descent method according to the cache pool; and whether the training round number Episode reaches a round iteration number threshold is judged, to obtain a first judgment result;

[0023] If the first judgment result is no, whether the first iteration number i is an integer multiple of the training flexible workshop environment is judged, to obtain a second judgment result;

[0024] If the second determination result is no, the current training flexible workshop environment is updated, the value of the first iteration number i is increased by 1, and the step of determining the process routing and the machine competition graph of the current training flexible workshop environment is returned to;

[0025] If the second determination result is yes, the parameters of the updated feature extraction network, the parameters of the updated Actor network, and the parameters of the updated Critic network are determined as the to-be-determined network parameters of the Episode training round;

[0026] According to the buffer pool, the current strategy is determined, and the current strategy is verified by using the plurality of verification sets to determine the evaluation quantity of the Episode training round;

[0027] According to the evaluation quantity of the Episode training round, the evaluation quantity of the Episode-1 training round, the to-be-determined network parameters of the Episode training round, and the network parameters of the Episode-1 training round, the network parameters of the Episode training round are determined, the training flexible workshop environment set is updated, the value of the training round number Episode is increased by 1, the value of the first iteration number i is increased by 1, and the step of determining any training flexible workshop environment in the training flexible workshop environment set as the current training flexible workshop environment is returned to;

[0028] If the first determination result is yes, it is determined that the feature extraction network, the Actor network, and the Critic network are all optimized, and the maximum completion time corresponds to the optimal scheduling plan.

[0029] Optionally, the process routing is used to describe the process that can be completed in the flexible workshop environment, the process characteristics that can be completed in the flexible workshop environment, the process processing sequence of the same workpiece, and the sequence of multiple processes completed on the same machine when producing the same workpiece.

[0030] The process characteristics include the scheduling mark of the process in the current state, the estimated lower bound of the completion time, the processing time span, the average processing time, the queuing time, the number of remaining processes of the workpiece, the remaining work amount of the workpiece, and the number of processable machines.

[0031] Optionally, the machine competition graph includes multiple machines in the flexible workshop environment, machine characteristics of the multiple machines in the flexible workshop environment, and competition relationships of the multiple machines.

[0032] The machine characteristics include the number of processable candidates of the machine in the current state, the total number of processable processes, the average processing time, the queuing time, the idle time, and the current queue length.

[0033] Optionally, the updating the buffer pool according to the process map and the machine competition map, using the feature extraction network, the Actor network and the Critic network comprises:

[0034] initializing state information of the 0th iteration;

[0035] initializing a process feature map and a machine feature map;

[0036] setting a second iteration number t as 1;

[0037] obtaining state information of the (t-1)th iteration;

[0038] inputting the state information of the (t-1)th iteration into the feature extraction network to obtain process features and machine features;

[0039] updating the process feature map using the process features;

[0040] updating the machine feature map using the machine features;

[0041] inputting the process feature map and the machine feature map into the Actor network to obtain a scheduling strategy of the (t-1)th iteration;

[0042] sampling the scheduling strategy of the (t-1)th iteration to obtain an action of the (t-1)th iteration;

[0043] interacting the action of the (t-1)th iteration with a current training flexible workshop environment to obtain an iteration reward and state information of the tth iteration;

[0044] inputting the process feature map and the machine feature map into the Critic network to obtain an advantage function value;

[0045] adding the state information of the (t-1)th iteration, the action of the (t-1)th iteration, the iteration reward, the state information of the tth iteration and the advantage function value into the buffer pool;

[0046] judging whether the second iteration number t reaches a total number of processes of the current training flexible workshop to obtain a third judgment result;

[0047] if the third judgment result is yes, adding the total number of processes of the current training flexible workshop as a termination marker into the buffer pool;

[0048] if the third judgment result is no, increasing the second iteration number t by 1 and returning to the step of obtaining the state information of the (t-1)th iteration.

[0049] Optionally, a current strategy is determined according to the cache pool, and the current strategy is verified by using a plurality of verification sets to determine the evaluation quantity of the Episode training round, comprising:

[0050] The current strategy is interacted with a plurality of verification flexible workshop environments in the verification flexible workshop environment set respectively to obtain a maximum completion time corresponding to each verification flexible workshop environment; the verification flexible workshop environment is a flexible workshop environment corresponding to the verification set;

[0051] The mean of the plurality of maximum completion times corresponding to the verification flexible workshop environment set is determined as the evaluation quantity of the Episode training round.

[0052] Optionally, the network parameters of the Episode training round are determined according to the evaluation quantity of the Episode training round, the evaluation quantity of the Episode-1 training round, the pending network parameters of the Episode training round and the network parameters of the Episode-1 training round, comprising:

[0053] It is determined whether the evaluation quantity of the Episode training round is greater than the evaluation quantity of the Episode-1 training round to obtain a fourth determination result;

[0054] If the fourth determination result is yes, the pending network parameters of the Episode training round are determined as the network parameters of the Episode training round;

[0055] If the fourth determination result is no, the network parameters of the Episode-1 training round are determined as the network parameters of the Episode training round.

[0056] A flexible workshop scheduling system, comprising:

[0057] A flexible workshop environment generation module is configured to randomly generate a plurality of flexible workshop environments in a production workshop according to a preset production target; a plurality of data sets are constructed according to parameters of the plurality of flexible workshop environments; the data sets correspond one-to-one to the flexible workshop environments; the data sets are training sets or verification sets;

[0058] A scheduling strategy model determination module is configured to construct a scheduling strategy model of the production workshop based on a Markov decision process;

[0059] A network parameter initialization module is configured to perform parameter initialization processing on a feature extraction network, an Actor network and a Critic network;

[0060] An optimal scheduling plan determination module is configured to optimize the feature extraction network, the Actor network and the Critic network by using the scheduling strategy model and the multiple data sets, and determine the scheduling plan corresponding to the maximum completion time as the optimal scheduling plan after the optimization is completed.

[0061] A scheduling module is configured to complete the preset production target based on the optimal scheduling plan.

[0062] An electronic device includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the flexible job shop scheduling method.

[0063] Optionally, the memory is a readable storage medium.

[0064] According to the embodiments of the present application, the following technical effects are provided:

[0065] The flexible job shop scheduling method, system and electronic device provided by the present application include the following steps: generating multiple flexible job shop environments in a production workshop according to a preset production target; constructing multiple data sets according to parameters of the multiple flexible job shop environments; the data sets correspond to the flexible job shop environments one by one; the data sets are training sets or validation sets; constructing a scheduling strategy model of the production workshop based on a Markov decision process; performing parameter initialization processing on a feature extraction network, an Actor network and a Critic network; optimizing the feature extraction network, the Actor network and the Critic network by using the scheduling strategy model and the multiple data sets, and determining the scheduling plan corresponding to the maximum completion time as the optimal scheduling plan after the optimization is completed; and completing the preset production target based on the optimal scheduling plan. The present application extracts features from multiple workshop environments to generate a scheduling scheme, thereby improving the efficiency and rationality of flexible job shop scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0067] Figure 1 A flexible job shop scheduling method flow chart is provided by the present application.

[0068] Figure 2 A process processing diagram is provided by the present application.

[0069] Figure 3A machine competition schematic diagram provided by the present application;

[0070] Figure 4 A feature extraction model based on a graph neural network provided by the present application;

[0071] Figure 5 A Markov decision model state transition process schematic diagram provided by the present application;

[0072] Figure 6 A flexible job shop scheduling method principle diagram provided by the present application;

[0073] Figure 7 A production Gantt chart provided by the present application;

[0074] Figure 8 A running effect diagram on a 3x3x9 scale flexible job shop environment provided by the present application;

[0075] Figure 9 A running effect diagram on a 6x6x36 scale flexible job shop environment provided by the present application;

[0076] Figure 10 A running effect diagram on a 10x5x50 scale flexible job shop environment provided by the present application;

[0077] Figure 11 A running effect diagram on a 20x5x100 scale flexible job shop environment provided by the present application. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0079] The purpose of the present application is to provide a flexible job shop scheduling method, system and electronic device, which can respectively perform feature extraction on multiple job shop environments for generating scheduling schemes, thereby improving the efficiency and rationality of flexible job shop scheduling.

[0080] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0081] Embodiment 1

[0082] As shown in the figure, the present embodiment provides a flexible job shop scheduling method, comprising: Figure 1

[0083] ​Step 101: randomly generating a plurality of flexible workshop environments in the production workshop according to a preset production target; constructing a plurality of data sets according to parameters of the plurality of flexible workshop environments; the data sets correspond one-to-one to the flexible workshop environments; the data sets are training sets or verification sets;

[0084] Step 102: constructing a scheduling strategy model of the production workshop based on a Markov decision process;

[0085] Step 103: performing parameter initialization processing on the feature extraction network, the Actor network and the Critic network;

[0086] Step 104: simultaneously optimizing the feature extraction network, the Actor network and the Critic network by using the scheduling strategy model and the plurality of data sets, and determining a scheduling plan corresponding to a maximum completion time as an optimal scheduling plan after the optimization is completed;

[0087] Step 104, comprising:

[0088] Step 1041: determining the initialized feature extraction network parameters, the Actor network parameters and the Critic network parameters as network parameters of the 0th training round.

[0089] Step 1042: initializing a critic quantity of the 0th training round.

[0090] Step 1043: letting a training round number Episode = 1.

[0091] Step 1044: initializing a cache pool and a capacity of the scheduling strategy model.

[0092] Step 1045: letting a first iteration number i = 1.

[0093] Step 1046: determining any training flexible workshop environment in a training flexible workshop environment set as a current training flexible workshop environment; the training flexible workshop environment is a flexible workshop environment corresponding to a training set.

[0094] Step 1047: determine the process processing graph and the machine competition graph of the current training flexible workshop environment. The process processing graph is used to describe the process that can be completed in the flexible workshop environment, the process characteristics that can be completed in the flexible workshop environment, the process processing sequence of the same workpiece, and the sequence of multiple processes completed on the same machine when producing the same workpiece; wherein the process characteristics include the scheduling mark of the process in the current state, the estimated lower bound of the completion time, the processing time span, the average processing time, the queuing time, the number of remaining processes of the workpiece, the remaining work amount of the workpiece, and the number of processable machines. The machine competition graph includes multiple machines in the flexible workshop environment, machine characteristics of multiple machines in the flexible workshop environment, and competition relationship of multiple machines; the machine characteristics include the number of candidates that can be processed by the machine in the current state, the total number of processable processes, the average processing time, the queuing time, the idle time, and the current queue length.

[0095] Step 1048: according to the process processing graph and the machine competition graph, using the feature extraction network, the Actor network and the Critic network, update the cache pool.

[0096] Step 1048 includes:

[0097] Step 10481: initialize the state information of the 0th iteration.

[0098] Step 10482: initialize the process feature graph and the machine feature graph.

[0099] Step 10483: let the second iteration number t = 1.

[0100] Step 10484: obtain the state information at the t-1th iteration.

[0101] Step 10485: input the state information at the t-1th iteration into the feature extraction network to obtain the process feature and the machine feature.

[0102] Step 10486: update the process feature graph using the process feature.

[0103] Step 10487: update the machine feature graph using the machine feature.

[0104] Step 10488: input the process feature graph and the machine feature graph into the Actor network to obtain the scheduling strategy at the t-1th iteration.

[0105] Step 10489: sample the scheduling strategy at the t-1th iteration to obtain the action generated at the t-1th iteration.

[0106] Step 104810: interact the action at the t-1th iteration with the current training flexible workshop environment to obtain the iteration reward and the state information at the tth iteration.

[0107] Step 104811: input the process feature map and the machine feature map into the Critic network to obtain the advantage function value.

[0108] Step 104812: add the state information at the t-1th iteration, the action at the t-1th iteration, the iteration reward, the state information at the tth iteration and the advantage function value into the cache pool.

[0109] Step 104813: determine whether the second iteration number t reaches the total number of processes of the current training flexible workshop, and obtain a third determination result.

[0110] Step 104814: if the third determination result is yes, add the total number of processes of the current training flexible workshop as a termination marker to the cache pool.

[0111] Step 104815: if the third determination result is no, increase the value of the second iteration number t by 1, and return to the step of "obtaining the state information at the t-1th iteration".

[0112] Step 1049: according to the cache pool, update the parameters of the feature extraction network, the Actor network and the Critic network by using the gradient descent method; and determine whether the training round number Episode reaches the iteration threshold, to obtain a first determination result.

[0113] Step 10410: if the first determination result is no, determine whether the first iteration number i is an integer multiple of the training flexible workshop environment, to obtain a second determination result.

[0114] Step 10411: if the second determination result is no, update the current training flexible workshop environment, increase the value of the first iteration number i by 1, and return to the step of "determining the process processing map and the machine competition map of the current training flexible workshop environment".

[0115] Step 10412: if the second determination result is yes, determine the updated parameters of the feature extraction network, the updated parameters of the Actor network and the updated parameters of the Critic network as the to-be-determined network parameters of the Episode training round.

[0116] Step 10413: determine the current policy according to the cache pool, and verify the current policy by using a plurality of verification sets to determine the evaluation quantity of the Episode training round.

[0117] Step 10413, comprising:

[0118] Step 104131: interact the current strategy with multiple verification flexible workshop environments in the verification flexible workshop environment set respectively, to obtain a maximum completion time corresponding to each verification flexible workshop environment; the verification flexible workshop environment is the flexible workshop environment corresponding to the verification set.

[0119] Step 104132: determine the average of the multiple maximum completion times corresponding to the verification flexible workshop environment set as the evaluation quantity of the Episode training round.

[0120] Step 10414: determine the network parameters of the Episode training round according to the evaluation quantity of the Episode training round, the evaluation quantity of the Episode-1 training round, the pending network parameters of the Episode training round and the network parameters of the Episode-1 training round; update the training flexible workshop environment set; increase the value of the training round number Episode by 1, increase the value of the first iteration number i by 1, and return to the step “determine that any training flexible workshop environment in the training flexible workshop environment set is the current training flexible workshop environment”.

[0121] Step 10414, comprising:

[0122] Step 104141: determine whether the evaluation quantity of the Episode training round is greater than the evaluation quantity of the Episode-1 training round, to obtain a fourth determination result;

[0123] Step 104142: if the fourth determination result is yes, determine that the pending network parameters of the Episode training round are the network parameters of the Episode training round;

[0124] Step 104143: if the fourth determination result is no, determine that the network parameters of the Episode-1 training round are the network parameters of the Episode training round.

[0125] Step 10415: if the first determination result is yes, determine that the feature extraction network, the Actor network and the Critic network are all optimized, and determine that the scheduling plan corresponding to the maximum completion time is the optimal scheduling plan.

[0126] Step 105: complete the preset production target based on the optimal scheduling plan.

[0127] Embodiment 2

[0128] The flexible workshop environment graph modeling method provided in the application is as shown in Figures 2-3 The process processing graph and the machine competition graph are used to respectively describe the process and the machine. The process processing graph is defined as G J = <V J , EJ , F J >, where V J It's G J The node set of E corresponds to the process in the workshop; J It's G J The edge set consists of two types of edges: one type is used to represent the processing sequence of processes belonging to the same workpiece, and the other type is used to represent the processing sequence of processes on the same machine; F J It's G J The feature set of the middle node, the feature of each process is represented by an 8-dimensional vector, which is the scheduling mark of the process in the current state, the estimated lower bound of the completion time, the processing time span, the average processing time, the queue time, the number of remaining processes for the workpiece, the remaining workload of the workpiece, and the number of machines that can be processed. The machine competition graph is defined as G M = Where V M It's G M The node set E M It's G M The edge set of F is used to connect the machines with competitive processing relationships using undirected edges; M It's G M The feature set of the middle node, the feature of each machine is represented by a 6-dimensional vector, which is the number of processable candidates, the total number of processable processes, the average processing time, the queue time, the idle time, and the current queue length of the machine in the current state; G M The edge feature set is defined as the sum of the features of the processes competed by the nodes at both ends of the edge. The flexible shop floor plan modeling method based on the above definition can fully describe the production unit information and shop floor structure information.

[0129] The feature extraction model based on graph neural network provided by the present invention is as follows Figure 4 As shown in the figure, the model uses graph attention network and edge feature graph attention network to learn process features and machine features respectively, and in the process of network iteration, the features of process and machine nodes are interacted according to the production relationship: at the lth layer of the network, the process node feature set of this layer is With G J The adjacency matrix A J Input to the GAT module GAT of this layer (l) , to obtain the next layer of process node feature set At the same time, according to G M The edge set E M and the current state of the scheduled process to calculate G M The edge feature map tensor of layer l and through and Calculate Then the machine node feature set of this layer G M The adjacency matrix A M and Input to the EGAT module EGAT of this layer (l) , to obtain the feature set of the machine nodes in the next layer The feature extraction model defined in this way can learn richer node features.

[0130] The state transition process in the Markov decision model of the scheduling task provided by the present invention is as follows: Figure 5 As shown in , after each step of the scheduling action is completed, the next scheduling time can be calculated based on the existing scheduling information, and the environment information is updated to that time. Specifically, for the process processing diagram, the process feature set F J It will be recalculated based on the environmental information, and for the machine selected in the current step, its previous processing step will be connected to the process selected in the current step, that is, a new edge will be generated in the process processing graph; for the machine competition graph, the machine feature set F M Will be recalculated according to the environment information. At the same time, since the competition relationship between machines has changed after the state update, the edge set E M The state transition process defined in this way fully utilizes the dynamically changing relationships between nodes during the production process, which is beneficial for the agent to perceive the environment.

[0131] like Figure 6 As shown, an embodiment of the present invention provides a flexible workshop scheduling method based on dual-view graph reinforcement learning, comprising the following steps:

[0132] S1, randomly initialize the feature extraction network parameters ω, Actor network parameters θ and Critic network parameters . Randomly generate a flexible workshop environment of a specified size, including N train The training set D of the environment train and containing N vali The validation set D of the environment vali . Set the algorithm related hyperparameters: feature extraction network iteration number L fea , output node feature dimension d out , Actor network and Critic network layer number L agent and dimension d aqent wait.

[0133] S2, if the number of training times reaches the preset value T ep , then the training ends. Otherwise, each flexible workshop environment is transformed into a form based on the process processing diagram and the machine competition diagram: collect data for the i-th environment, generate a node set based on the number of processes and their workpieces and edge sets Generate feature sets based on production time and processing relationships and Then generate a node set based on the number of machines and the initial competition relationship Edge Set and edge feature sets The initial input state of the scheduling model is obtained

[0134] S3, let the agent explore each environment, and repeat the following operation T for the i-th environment i times(T i is the number of processes in the i-th environment): First, the state information of the previous moment Input to the feature extraction network to obtain the extracted process features and machine characteristics Secondly, the node features corresponding to each action are transformed from and Extract and connect the graph features and (The mean of each node feature) is input to the fully connected Actor network to generate the action strategy of the current step agent And according to Sampling generation action The agent then takes action Interact with the environment to obtain the current state Rewards for this step End mark for this step (If t=T i ); Finally, the graph features and Input to the fully connected Critic network to calculate the state value, and further calculate the advantage function value of the corresponding step save To the cache pool, let t←t-1.

[0135] S4, calculates the loss function of the PPO algorithm based on the data in the cache pool, which is defined as: In the formula π θ and are the actual update strategy and exploration strategy respectively, ∈ is the truncation coefficient; is the mean square error between the estimated state value and the cumulative reward; S[π θ ](s t ) is state s tThe entropy of the strategy; c1 and c2 are positive coefficients. Update the feature extraction network parameters ω, Actor network parameters θ and Critic network parameters based on the gradient descent method.

[0136] S5, judge the current number of training times: If the current number of training times is T vali The current strategy is verified on the validation set given by S1, that is, let the current strategy π θ and N in the validation set vali The scheduler interacts with each environment to generate a scheduling solution (maximum completion time). The average of the scheduler's scheduling solutions in these environments is used as the evaluation criterion. If the quality of the current solution is improved compared to the past, the parameters of the current network are saved. If the current number of training times is T sp If the value is an integer multiple of , the environment set for training will be regenerated.

[0137] S6, repeatedly perform S2 to S5 until the number of training times reaches the preset value T ep Finally, we get the model parameters ω, θ,

[0138] like Figure 7 As shown, Figure 7 A flexible shop scheduling method based on dual-view graph reinforcement learning provided in this embodiment is used to schedule the corresponding production Gantt chart in a 10×5×50 (representing 10 workpieces, 5 machines, and 50 processes, the same below) environment;

[0139] The operating effect diagram of an example of the flexible workshop scheduling method based on dual-view graph reinforcement learning in four flexible workshop environments of different sizes is shown in the following figure: Figures 8-11 As shown, the four different scales are: 3×3×9, 6×6×36, 10×5×50, and 20×5×100. The operation effect diagram shows the comparison of the results of the flexible workshop scheduling method based on dual-view graph reinforcement learning provided by the present invention and four classic heuristic rules. These four heuristic rules are: First In First Out (FIFO), Most Operation Remaining (MOR), Shortest Processing Time (SPT), and Most Work Remaining (MWKR). In the operation effect diagram, the flexible workshop scheduling method based on dual-view graph reinforcement learning provided by the present invention is superior to these heuristic rules in terms of solution quality, which fully reflects the effectiveness of the method.

[0140] Example 3

[0141] In order to perform the method corresponding to the above-mentioned embodiment one, to realize the corresponding functions and technical effects, a flexible workshop scheduling system is provided below, comprising:

[0142] A flexible workshop environment generation module is configured to generate a plurality of flexible workshop environments in the production workshop according to a preset production target; construct a plurality of data sets according to parameters of the plurality of flexible workshop environments; the data sets correspond one-to-one to the flexible workshop environments; and the data sets are training sets or validation sets.

[0143] A scheduling strategy model determination module is configured to construct a scheduling strategy model of the production workshop based on a Markov decision process.

[0144] A network parameter initialization module is configured to perform parameter initialization processing on the feature extraction network, the Actor network and the Critic network.

[0145] An optimal scheduling plan determination module is configured to simultaneously optimize the feature extraction network, the Actor network and the Critic network by using the scheduling strategy model and the plurality of data sets, and determine the scheduling plan corresponding to the maximum completion time as the optimal scheduling plan after the optimization is completed.

[0146] A scheduling module is configured to complete the preset production target based on the optimal scheduling plan.

[0147] Embodiment 4

[0148] The embodiment provides an electronic device, comprising a memory and a processor, the memory is used for storing a computer program, and the processor runs the computer program to make the electronic device execute the flexible workshop scheduling method of the embodiment 1 or 2. Wherein, the memory is a readable storage medium.

[0149] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0150] The principles and implementation modes of the present application are described by using specific examples in this paper. The above embodiment description is only used to help understand the method and core idea of the present application; at the same time, for the general technical personnel in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A flexible workshop scheduling method, characterized in that: include: Randomly generate multiple flexible workshop environments in the production workshop according to preset production goals; Construct multiple data sets based on parameters of multiple flexible workshop environments; The data set corresponds one-to-one to the flexible workshop environment; the data set is a training set or a validation set; Construct a scheduling strategy model for the production workshop based on the Markov decision process; Initialize the parameters of the feature extraction network, actor network, and critic network; Utilizing the scheduling strategy model and the plurality of data sets, the feature extraction network, the actor network, and the critic network are simultaneously optimized. After the optimization is completed, a scheduling plan corresponding to the maximum completion time is determined as the optimal scheduling plan, including: Determine the initialized feature extraction network parameters, Actor network parameters, and Critic network parameters as the network parameters for the 0th training round; Initialize the evaluation metric for the 0th training round; Set the number of training rounds Episode=1; Initialize the cache pool and capacity of the scheduling strategy model; Let the first iteration number i=1; Determine any training flexible workshop environment in the training flexible workshop environment set as the current training flexible workshop environment; the training flexible workshop environment is the flexible workshop environment corresponding to the training set; Determine the process diagram and machine competition diagram for the current training flexible workshop environment; According to the process graph and the machine competition graph, the cache pool is updated using the feature extraction network, the actor network, and the critic network; According to the cache pool, the parameters of the feature extraction network, the actor network, and the critic network are updated using the gradient descent method; and whether the number of training rounds Episode reaches the round iteration number threshold is determined to obtain a first judgment result; If the first judgment result is no, determine whether the first iteration number i is an integer multiple of the training flexible workshop environment to obtain a second judgment result; If the second judgment result is no, then the current training flexible workshop environment is updated, the value of the first iteration number i is increased by 1, and the process returns to step "determining the process diagram and machine competition diagram of the current training flexible workshop environment"; If the second judgment result is yes, the parameters of the updated feature extraction network, the parameters of the updated actor network, and the parameters of the updated critic network are determined as the pending network parameters for the Episode th training round; Determine the current strategy based on the cache pool, and use multiple verification sets to verify the current strategy, and determine the evaluation amount of the Episode th training round; Determine the network parameters of the Episode-th training round based on the evaluation value of the Episode-th training round, the evaluation value of the Episode-1th training round, the undetermined network parameters of the Episode-th training round, and the network parameters of the Episode-1th training round; update the training flexible workshop environment set; increase the value of the number of training rounds Episode by 1, increase the value of the first iteration number i by 1, and return to the step of "determining any training flexible workshop environment in the training flexible workshop environment set as the current training flexible workshop environment"; If the first judgment result is yes, it is determined that the feature extraction network, the actor network, and the critic network are all optimized, and the scheduling plan corresponding to the maximum completion time is determined to be the optimal scheduling plan; The preset production target is completed based on the optimal scheduling plan.

2. A flexible workshop scheduling method according to claim 1, characterized in that: The process diagram is used to describe the processes that can be completed in the flexible workshop environment, the characteristics of the processes that can be completed in the flexible workshop environment, the process processing sequence of the same workpiece, and the sequence of multiple processes completed on the same machine when producing the same workpiece; Among them, the process characteristics include the scheduling mark of the process in the current state, the estimated lower bound of the completion time, the processing time span, the average processing time, the queue time, the number of remaining processes for the workpiece, the remaining workload of the workpiece and the number of machines that can be processed.

3. A flexible workshop scheduling method according to claim 1, characterized in that: The machine competition graph includes a plurality of machines in the flexible workshop environment, machine features of the plurality of machines in the flexible workshop environment, and competition relationships among the plurality of machines; The machine characteristics include the number of processable candidates, the total number of processable processes, the average processing time, the queue time, the idle time and the current queue length of the machine in the current state.

4. A flexible workshop scheduling method according to claim 1, characterized in that: The updating of the cache pool according to the process processing graph and the machine competition graph by using the feature extraction network, the actor network, and the critic network includes: Initialize the state information of the 0th iteration; Initialize process feature map and machine feature map; Let the second iteration number t=1; Get the status information at the t-1th iteration; Input the state information at the t-1th iteration into the feature extraction network to obtain process features and machine features; Updating the process characteristic graph using the process characteristic; updating the machine feature map using the machine feature; Input the process feature graph and the machine feature graph into the Actor network to obtain the scheduling strategy at the t-1th iteration; Sample the scheduling strategy at the t-1th iteration to generate the action at the t-1th iteration; Use the action at the t-1th iteration to interact with the current training flexible workshop environment to obtain the iteration reward and the state information at the tth iteration; Inputting the process characteristic graph and the machine characteristic graph into the Critic network to obtain an advantage function value; Adding the state information at the t-1th iteration, the action at the t-1th iteration, the iteration reward, the state information at the tth iteration, and the advantage function value to the cache pool; Determine whether the second iteration number t reaches the total number of processes of the current training flexible workshop, and obtain a third determination result; If the third judgment result is yes, the total number of processes of the current training flexible workshop is added to the buffer pool as a termination mark; If the third judgment result is no, the value of the second iteration number t is increased by 1, and the process returns to step "obtaining the status information at the t-1th iteration".

5. The flexible workshop scheduling method according to claim 1, characterized in that: Determine the current strategy based on the cache pool, and use multiple validation sets to verify the current strategy, and determine the evaluation metric for the Episode th training round, including: Interacting the current strategy with multiple verification flexible workshop environments in the verification flexible workshop environment set to obtain the maximum completion time corresponding to each verification flexible workshop environment; the verification flexible workshop environment is the flexible workshop environment corresponding to the verification set; The mean of multiple maximum completion times corresponding to the flexible workshop environment set is determined as the evaluation metric for the Episode th training round.

6. A flexible workshop scheduling method according to claim 1, characterized in that: Determining the network parameters of the Episode th training round according to the evaluation amount of the Episode th training round, the evaluation amount of the Episode th-1 th training round, the undetermined network parameters of the Episode th training round, and the network parameters of the Episode th-1 th training round includes: Determine whether the evaluation value of the Episode-th training round is greater than the evaluation value of the Episode-1-th training round, and obtain a fourth judgment result; If the fourth judgment result is yes, determining the undetermined network parameters of the Episode th training round as the network parameters of the Episode th training round; If the fourth judgment result is no, the network parameters of the Episode-1 th training round are determined as the network parameters of the Episode th training round.

7. A flexible workshop scheduling system, characterized in that: The flexible workshop scheduling system applies the flexible workshop scheduling method according to any one of claims 1 to 6, and the flexible workshop scheduling system includes: A flexible workshop environment generation module is used to randomly generate multiple flexible workshop environments in the production workshop according to preset production targets; construct multiple data sets based on the parameters of the multiple flexible workshop environments; the data sets correspond one-to-one to the flexible workshop environments; the data sets are training sets or validation sets; Scheduling strategy model determination module, used to build a scheduling strategy model for the production workshop based on the Markov decision process; The network parameter initialization module is used to initialize the parameters of the feature extraction network, actor network and critic network; an optimal scheduling plan determination module, configured to utilize the scheduling strategy model and the plurality of data sets to simultaneously optimize the feature extraction network, the actor network, and the critic network, and determine, after the optimization is complete, that the scheduling plan corresponding to the maximum completion time is the optimal scheduling plan; A scheduling module is used to complete the preset production target based on the optimal scheduling plan.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a flexible workshop scheduling method according to any one of claims 1 to 6.

9. The electronic device according to claim 8, characterized in that: The memory is a readable storage medium.

Images