Flexible workshop joint scheduling method and device based on graph neural network and reinforcement learning

Abstract

The present invention belongs to the field of discrete manufacturing technology, and provides a flexible workshop joint scheduling method and device based on graph neural network and reinforcement learning, which can fully describe the constraint relationship between processing and transportation of flexible workshops with transportation resource constraints, and effectively improve the scheduling quality of decision-making models. The process of the method is: constructing a scheduling agent model; training the scheduling agent model, including the design of states, actions, state transitions and reward functions; based on the feature extraction network and Actor network of the trained agent model, realizing flexible workshop joint scheduling, and outputting the joint scheduling actions of processes, machines and unmanned vehicles.

Description

Technical Field

[0001] The present invention belongs to the technical field of discrete manufacturing, and in particular relates to a flexible workshop joint scheduling method and device based on graph neural network and reinforcement learning. Background Art

[0002] In the manufacturing industry, the manufacturing capabilities of enterprises are closely linked to the resource allocation strategies they adopt. Faced with the current intensified market competition and the diversification of customer needs, it is urgent to establish a scheduling system with real-time, wide applicability, flexibility and scalability to arrange production tasks, so as to achieve the optimal configuration of production resources and maximize production benefits. Therefore, the research on intelligent optimization scheduling and autonomous decision-making technology in discrete manufacturing industry not only has far-reaching theoretical significance, but also has significant economic application value. The Flexible Job co-Scheduling Problem (FJCSP) is a generalization and refinement of the Flexible Job Shop Scheduling Problem (FJSP). It not only considers the processing of workpieces by machines, but also considers the use of unmanned vehicles (AGVs) to transport workpieces. This problem is very close to the actual production environment's demand for production flexibility and diversity, and therefore has received widespread attention from academia and industry.

[0003] FJCSP is usually modeled as a combinatorial optimization problem, and it has been proven to be an NP-hard problem. This means that the use of traditional scheduling methods, mainly including exact methods, metaheuristic methods and heuristic methods, has certain application bottlenecks when solving FJCSP problems. For example, exact methods such as constraint programming and mixed integer programming can find the optimal solution to the original problem, but usually have exponential computational complexity and cannot meet the real-time scheduling requirements of actual production scenarios; metaheuristic methods such as genetic algorithms, particle swarm optimization algorithms and ant colony optimization algorithms are widely used in engineering, but their performance is sensitive to parameters and has poor generalization, and they also face the problem of not being able to be applied to real-time scheduling; heuristic methods refer to a class of methods that use prior knowledge to pre-process problems and find solutions through preset rules. These methods are relatively simple in program design and have good real-time scheduling capabilities and universal applicability. However, the quality of the solutions obtained by such methods is usually not optimal, and most of them can only be applied to certain specific scenarios.

[0004] In recent years, deep reinforcement learning methods have shown advantages in many fields, including combinatorial optimization represented by scheduling problems and traveling salesman problems. In the workshop scheduling task, the reinforcement learning method models the task as a Markov decision process (MDP), allowing the agent to conduct extensive exploration and learning in the simulated workshop environment. It is a data-driven method that can be applied in an offline environment. When the agent applies the strategy to make decisions in the actual workshop environment, the scheduling result can be given in a very short time. Therefore, this method has the advantages of both data learning and real-time decision-making, and effectively overcomes the shortcomings of traditional methods in terms of reliance on prior knowledge and lack of real-time performance. However, although reinforcement learning has been successfully applied to the solution of multiple combinatorial optimization problems, its application in the field of FJCSP is still relatively small. Nazari et al. (Real-time data-driven dynamic scheduling for flexible jobshop with insufficient transportation resources using hybrid deep q network,” Robotics and Computer-Integrated Manufacturing, vol. 74, p. 102283, 2022.) proposed a deep Q-network (DQN)-based method to simultaneously optimize the completion time and energy consumption in FJSPT. However, the action space they designed is a combination of a series of scheduling rules, which hinders the full exploration of the solution space. Mayer et al. (Modular production control using deep reinforcement learning: proximal policy optimization, Journal of Intelligent Manufacturing, vol. 32, no. 8, pp. 2335–2351, 2021.) designed an end-to-end deep reinforcement learning method for solving FJCSP, but its Markov model and decision mechanism make the model only applicable to fixed-size instances and cannot be migrated to problems of different scales. Moon et al. (Learning-enabled flexible job-shop scheduling for scalable smartmanufacturing, arXiv preprint,arXiv:2402.08979, 2024.)Designed a reinforcement learning algorithm that uses complex heterogeneous disjunctive graphs and demonstrated the ability to generalize to large scale.Nevertheless, they used a complex encoder-decoder network combined with a policy gradient (PG) algorithm for training, which resulted in prolonged training time and a large number of iterations, thus losing the time advantage of the reinforcement learning scheduling method.

[0005] Although the above scheduling solution methods can be used to solve the flexible workshop scheduling problem with transportation resource constraints, there are some problems, mainly manifested in insufficient exploration of the solution space, low computational efficiency, lack of generalization and migration capabilities, etc. Therefore, it is urgent to propose a new intelligent scheduling decision method to solve the above problems and improve the intelligence, autonomy and practicality of the flexible workshop scheduling decision model with transportation resource constraints. Summary of the invention

[0006] The present invention provides a flexible workshop joint scheduling method and device based on graph neural network and reinforcement learning, which can comprehensively describe the constraint relationship between processing and transportation of flexible workshops with transportation resource constraints, and effectively improve the scheduling quality of the decision-making model.

[0007] The technical solution for implementing the present invention is as follows:

[0008] In the first aspect, the embodiment of the present application provides a flexible workshop joint scheduling method based on graph neural network and reinforcement learning, and the specific process is as follows:

[0009] Constructing a scheduling agent model, wherein the scheduling agent model includes a feature extraction network, an actor network, and a critic network; the feature extraction network includes a graph neural network and a fully connected neural network;

[0010] Training the scheduling agent model: input the state information of the previous moment into the feature extraction network to obtain the extracted environmental features; input the environmental features into the Actor network to generate the action strategy of the agent at the current step, and generate the scheduling action based on the action strategy sampling; execute the scheduling action in the current environment to obtain the state at the current moment and the reward for executing this action, and calculate the reward value fed back by the environment to the agent based on the state; input the global features into the fully connected Critic network to calculate the state value; calculate the loss function based on the state value and the reward value, and update the parameters of the agent model based on the loss function;

[0011] Based on the feature extraction network and Actor network of the trained intelligent agent model, flexible workshop joint scheduling is realized, and the joint scheduling actions of processes, machines and unmanned vehicles are output.

[0012] Furthermore, the specific process of the method of the present invention is:

[0013] S1, build the scheduling agent model including feature extraction network, Actor network and Critic network, and adjust the feature extraction network parameters , Actor network parameters and Critic network parameters Initialize; obtain flexible workshop environment data of different scales as training sets and test sets;

[0014] S2, set the upper limit of training times to , if the number of training times reaches the preset value , then the training ends, otherwise, each flexible workshop environment is initialized: The data of each environment is converted into the original features of the workshop environment, and the initial input state information of the scheduling agent model is obtained. , ;

[0015] S3: Let the scheduling agent model interact with each environment in parallel. Repeat the following steps for each environment times, the For the Number of processes in an environment:

[0016] First, the state information of the previous moment Input to the feature extraction network to obtain the extracted environmental features;

[0017] Secondly, the environmental features are input into the Actor network to generate the action strategy of the current agent , and the action strategy Sampling generates scheduling actions ;

[0018] Then, execute the scheduling action in the current environment , get the current state and the reward for performing this action ;

[0019] Finally, the global environment features are input into the Critic network to generate the state value , based on the state value, further calculate the advantage function value of the corresponding step , save the state, action, reward and advantage function to the experience buffer;

[0020] S4, based on the data stored in the experience buffer, calculate the loss function of the PPO algorithm, and use the loss function to update the feature extraction network parameters , Actor network parameters and Critic network parameters ;

[0021] S5, determine the current number of training times: If the current number of training times is The currently trained scheduling agent model is verified on the given validation set. If the average completion time of the currently generated scheduling action is less than the previous minimum completion time, the parameters of the current network are saved. If the current training times are If it is an integer multiple of , the environment set for training is regenerated;

[0022] S6, repeatedly perform S2 to S5 until the number of training times reaches the preset value , and finally obtain the model parameters optimized for the current scale problem , , , that is, the optimized scheduling agent model is obtained, and the optimized scheduling agent model is used to realize flexible workshop joint scheduling.

[0023] Furthermore, the flexible workshop environment data of the present invention includes the processing time of the process on different machines and the transportation time between machines.

[0024] Furthermore, the environmental features described in the present invention include: original features in the state information and features extracted by a feature extraction network, specifically: extracted process features and machine features, original unmanned vehicle features in the state information, and process-machine-unmanned vehicle combination features.

[0025] Furthermore, the graph neural network of the present invention is an edge feature graph attention network GAL, and the process of acquiring the environmental features is:

[0026] First, the original process features are input into the fully connected neural network MLP to obtain the extracted process features;

[0027] Secondly, the process features are summed up according to the graph relationship as the edge features of the graph, and are input into the edge feature graph attention network GAL together with the original machine features to obtain the process features and machine features;

[0028] Finally, the extracted process features, machine features, original unmanned vehicle features, and process-machine-unmanned vehicle triple combination features are concatenated as environmental features and used as input features for the subsequent Actor network.

[0029] Furthermore, the reward function of the present invention is:

[0030] ,

[0031] in, It is a process exist An estimate of the lower bound on the makespan for the time step.

[0032] Furthermore, the advantage function value of the present invention is for:

[0033]

[0034] in, and is the set hyperparameter, is the number of processes to be performed, It is the state value output by the Critic network.

[0035] Furthermore, the loss function of the present invention is:

[0036]

[0037]

[0038] is the mean square error between the estimated state value and the cumulative reward; Status The entropy of the next strategy, expect; , is a positive coefficient, and are the actual update strategy and the exploration strategy respectively. is the cutoff coefficient; is the advantage function value.

[0039] Furthermore, the present invention uses the loss function to update the feature extraction network parameters , Actor network parameters and Critic network parameters for:

[0040]

[0041]

[0042]

[0043] in, , is the learning rate for gradient descent.

[0044] In a second aspect, the embodiment of the present application is a flexible workshop joint scheduling device based on graph neural network and reinforcement learning, which is a scheduling agent model, including a feature extraction network and an Actor network;

[0045] The feature extraction network is used to extract features from the state information to obtain extracted environmental features, including a graph neural network and a fully connected neural network;

[0046] The fully connected neural network is used to receive the original process features and perform feature extraction to obtain process features, and add the process features according to the graph relationship as the edge features of the graph;

[0047] The graph neural network is used to extract features from the edge features of the graph and the original features of the machine to obtain process features and machine features;

[0048] The extracted process features, machine features, original unmanned vehicle features, and process-machine-unmanned vehicle triple combination features are concatenated as environmental features;

[0049] The Actor network is used to generate the action strategy of the current step agent based on the input environmental characteristics and generate scheduling actions for the samples.

[0050] Beneficial effects:

[0051] First, the flexible workshop joint scheduling method and device based on graph neural network and reinforcement learning can fully describe the constraint relationship between processing and transportation of flexible workshops with transportation resource constraints, so as to complete the complex problem of joint scheduling. In this method, the original feature design of production units such as processes, machines and unmanned vehicles contains rich dynamic information, which effectively improves the scheduling quality of the decision model.

[0052] Second, the feature extraction model proposed in the present invention, on the one hand, interacts each node according to the production relationship, so as to mine deeper feature information; on the other hand, it directly extracts key features from the environment for downstream decision-making tasks.

[0053] Third. The flexible workshop scheduling task for transportation resource constraints provided by the present invention includes the design of state, action, state transition and reward function. Regarding state design, the system state will include original features and extracted graph features, specifically: extracted process features and machine features, original unmanned vehicle features and process-machine-unmanned vehicle combination features; regarding action design, processing and transportation tasks will be considered at each decision step, and processes, machines and unmanned vehicles will be selected at the same time, so that the scheduling algorithm has a complete decision space for the original problem; regarding state transition, the environment will update the process, machine, unmanned vehicle features and combination features when receiving new actions; regarding reward function, the single-step reward is set as the lower bound difference between the two steps of the maximum process completion time. Under this design, pursuing the maximization of cumulative returns is equivalent to pursuing the minimization of order processing completion time. Therefore, such a reward function design is consistent with the actual goal to be optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 It is a schematic diagram of the interaction between the decision-making agent and the environment in the flexible workshop joint scheduling method based on graph neural network and reinforcement learning provided by the present invention;

[0056] Figure 2 It is a schematic diagram of a feature extraction model based on a graph neural network and a fully connected network in the flexible workshop joint scheduling method based on a graph neural network and reinforcement learning provided by the present invention;

[0057] Figure 3 It is a flow chart of model training in the flexible workshop joint scheduling method based on graph neural network and reinforcement learning provided by the present invention;

[0058] Figure 4 It is a diagram of the scheduling model training results in the flexible workshop joint scheduling method based on graph neural network and reinforcement learning provided by the present invention;

[0059] Figure 5 It is a production Gantt chart in one embodiment of the flexible workshop joint scheduling method based on graph neural network and reinforcement learning provided by the present invention;

[0060] Figure 6 This is a diagram showing the operating effects of the flexible workshop joint scheduling method based on graph neural network and reinforcement learning provided by the present invention in four simulation environments of different scales. DETAILED DESCRIPTION

[0061] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0062] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0063] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0064] like Figure 1 As shown, the method of interaction between the scheduling agent and the environment provided in the present invention includes a scheduling agent (scheduling agent) and a simulation environment (FJSPT environment).

[0065] Scheduling agent, including: feature extraction network (parameters are ), Actor network (parameters are ) and Critic network (parameters are ).

[0066] like Figure 2 As shown, the workshop simulation environment includes machines and unmanned vehicles. The machines perform processing according to the procedures, and different machines are scheduled and transferred by unmanned vehicles. The flexible workshop environment data includes process characteristics, machine characteristics, unmanned vehicle characteristics, and process-machine-unmanned vehicle triple characteristics.

[0067] like Figure 3 As shown, an embodiment of the present invention provides a flexible workshop joint scheduling method based on graph neural network and reinforcement learning, comprising the following steps:

[0068] S1, construct a simulated workshop environment model, and sample a batch of training data sets and a fixed verification data set based on the processes, machines and unmanned vehicles included in the environment model; the training data set and the verification data set include process features, machine features, unmanned vehicle features and combined features of triples (process-machine-unmanned vehicle), and the combined features reflect the processing time of the process on different machines and the transportation time between machines.

[0069] The workshop joint scheduling simulation environment provided in this embodiment is a simulation program based on Python language, including environment state initialization, environment state update and other parts. The main features are: 1. Allowing each job to have a different number of processes; 2. The number of jobs, processes, machines, and unmanned vehicles in the environment can be changed; 3. Supporting parallel simulation of multiple environments. The simulation environment can provide the scheduling agent with dynamic information of the current environment for subsequent decision-making.

[0070] Construct a scheduling agent model and initialize the model parameters. The agent model includes a feature extraction network, an Actor network, and a Critic network. The initialized model parameters include: feature extraction network parameters , Actor network parameters and Critic network parameters .

[0071] S2, set the upper limit of training times to , determine whether the number of training times reaches the preset value If yes, the training ends. Otherwise, the training data set is used to initialize the workshop environment: The data of each environment is used to calculate the original characteristics of the environment, and thus the initial input state of the scheduling agent model is obtained , .

[0072] S3, during the training process, the agent model uses the current strategy (i.e., the current model parameters) to interact with multiple environments in parallel to achieve the agent's exploration of multiple environments; the interaction is: selecting the original features of the entity corresponding to the current environment state and the combined features of the triplet (i.e., the state ) is input to the agent, the agent obtains the environmental features through the feature extraction network, the environmental features are input to the Actor network to obtain the probabilities of different joint actions (process-machine-unmanned vehicle triples), and the triple joint scheduling action is obtained through sampling / greedy strategies , perform actions in a workshop environment After that, the environment status is changed by Transition to , and the environment feeds back reward values ​​to the agent ; Input global features into the fully connected Critic network to calculate the state value ; Calculate the loss function based on the state value, and update the parameters of the intelligent agent model based on the loss function.

[0073] The following is a detailed description of the process. Interaction with the environment, Existing in the environment The following steps need to be repeated Second-rate:

[0074] First, the previous moment state information Input to the feature extraction network to obtain the extracted process features , Machine Features and unmanned vehicle features .

[0075] like Figure 2 As shown, the feature extraction network of this embodiment is implemented based on the feature extraction network of the graph neural network and the fully connected neural network. The network adopts the fully connected neural network (MLP) and the edge feature graph attention network (GAL) to learn the process features and machine features respectively, and in the process of network iteration, the process and machine features are interacted according to the production relationship. The specific process of this step is:

[0076] (1) In the network layer, the original features of the process of this layer Input into MLP to obtain the extracted process features.

[0077] (2) Add the process features according to the graph relationship as the edge features of the graph and compare them with the original machine features. Input them together into GAL for feature extraction.

[0078] (3) The process features after feature extraction Features of the machine and unmanned vehicle features, Original unmanned vehicle features and process-machine-unmanned vehicle triple combination features The splicing is used as the environment feature and input into the subsequent Actor network.

[0079] The feature extraction model based on the graph attention network and the fully connected neural network provided in this embodiment can effectively process the original node features provided by the simulation environment. The fully connected network model is used to learn the process features, and the edge feature graph attention network model is used to learn the machine features in the machine competition graph. In the process of network iterative learning of node features, the model supports the interaction between the process and the machine node according to the production relationship, so as to learn richer combination features.

[0080] Secondly, according to the combination characteristics corresponding to each process , and connect the extracted features , and Input to the fully connected Actor network to generate the action strategy of the current agent ,in is the discrete probability distribution of triple actions, for Sampling generates scheduling actions , the actions include the selection of processes, machines and unmanned vehicles.

[0081] Then, the actions are performed in the simulated workshop environment After that, the entity state at the current moment is calculated , the reward for this step, It is a process exist The estimated value of the lower bound of the completion time of the time step and the termination mark of this step (like ,otherwise );

[0082] Finally, the global features , , (The global feature is the average of the features extracted from all processes, machines, and unmanned vehicles) and is input into the fully connected Critic network to calculate the state value ;save to the cache pool, allowing time to move to the next decision moment .

[0083] Calculate the advantage function value:

[0084]

[0085] in, and is a hyperparameter, and Stored in the cache pool.

[0086] S4, calculates the loss function of the PPO algorithm based on the data in the cache pool, which is defined as:

[0087]

[0088]

[0089] and are the actual update strategy and the exploration strategy respectively. is the cutoff coefficient; is the mean square error between the estimated state value and the cumulative reward; Status The entropy of the next strategy, expect; , is a positive coefficient.

[0090] Update feature extraction network parameters based on gradient descent method , Actor network parameters and Critic network parameters ,in, , is the learning rate for gradient descent.

[0091] S5, determine the current number of training times: If the current number of training times is An integer multiple of , verify the current strategy (agent) on a fixed validation set, that is, let the current model And the validation set The agent interacts with the environment (only action selection is performed, but no training is performed. At this time, the feature extraction network and Actor network parameters in the agent are used, and the Critic network is not used. The Critic network is only used during training), generates scheduling actions and calculates the maximum completion time of scheduling actions. The average of the scheduling solutions of the agent in these environments is used as the judgment criterion. If the quality of the current solution is improved compared with the past, the parameters of the current network are saved; if the current number of training times is If the number is an integer multiple of , the environment set used for training is resampled.

[0092] In this embodiment, the number of training times (i.e., verification frequency) Set to 10, the number of training times (i.e. resampling frequency) Set to 20, the total number of training rounds Set to 1000.

[0093] S6, repeatedly perform S2 to S5 until the number of training times reaches the preset value Finally, we get the model parameters that are optimized for the current scale problem. , , , remember the optimal scheduling agent model obtained, and use the optimized scheduling agent model to realize flexible workshop joint scheduling.

[0094] In a second aspect, the embodiment of the present application is a flexible workshop joint scheduling device based on graph neural network and reinforcement learning, which is a scheduling agent model, including a feature extraction network and an Actor network;

[0095] The feature extraction network is used to extract features from the state information to obtain extracted environmental features;

[0096] The feature extraction network includes a graph neural network and a fully connected neural network;

[0097] The fully connected neural network MLP is used to receive the original process features and perform feature extraction to obtain process features, and add the process features according to the graph relationship as the edge features of the graph;

[0098] The edge feature graph attention network GAL is used to extract the edge features of the graph and the original features of the machine to obtain process features and machine features;

[0099] The extracted process features, machine features, original unmanned vehicle features, and process-machine-unmanned vehicle triple combination features are concatenated as environmental features;

[0100] Actor network is used to generate the action strategy of the current agent based on the input environment characteristics. , and generates scheduling actions for the samples.

[0101] The flexible workshop scheduling task for transportation resource constraints provided by the embodiment of the present application includes the design of states, actions, state transitions and reward functions. Regarding state design, the system state will include original features and extracted graph features, specifically: extracted process features and machine features, original unmanned vehicle features, and process-machine-unmanned vehicle combination features; regarding action design, processing and transportation tasks will be considered at each decision-making step, and processes, machines and unmanned vehicles will be selected at the same time, so that the scheduling algorithm has a complete decision space for the original problem; regarding state transfer, the environment will update the process, machine, unmanned vehicle features and combination features when receiving new actions; regarding the reward function, the single-step reward is set to the lower bound difference between the two steps of the maximum process completion time. Under this design, pursuing the maximization of cumulative returns is equivalent to pursuing the minimization of order processing completion time. Therefore, such a reward function design is consistent with the actual goal to be optimized.

[0102] The present invention adopts the Proximal Policy Optimization (PPO) algorithm based on the Actor-Critic framework to train the scheduling agent, and uses the Generalized Advantage Estimation (GAE) method to stabilize the training process.

[0103] like Figure 4 The figure shows the training result of the method of the present invention on an instance of scale 10x5x5 (representing 10 workpieces, 5 machines, and 5 unmanned vehicles, the same below). The blue curve in the figure is the scheduling result on the validation set during the model training process, which shows that the method can stably and quickly learn a good scheduling strategy.

[0104] like Figure 5As shown in FIG. 1 , the production Gantt chart corresponding to the scheduling solution of the present invention in a small-scale 2x2x2 example can intuitively display the current scheduling status and the final scheduling result. Due to the characteristics of the FJSPT problem itself, the vertical axis variable of the Gantt chart is the process, unlike the traditional FJSP Gantt chart where the vertical axis variable is the operation.

[0105] like Figure 6 As shown, it is an operation effect diagram of an example of the present invention in a flexible workshop environment of four different scales (10x5x2, 10x5x5, 20x5x5, 20x10x10). The operation effect diagram shows the comparison of the results of the flexible workshop scheduling method based on end-to-end reinforcement learning and transportation resource constraints provided by the present invention with four classic heuristic rules. The four heuristic rules are: First In First Out (FIFO), Most Operation Remaining (MOR), Shortest Processing Time (SPT) and Most Work Remaining (MWKR). In terms of vehicle scheduling, the strategy of using the first unmanned vehicle to perform the transportation task is adopted. In the operation effect diagram, the quality of the solution of the flexible workshop scheduling method based on end-to-end reinforcement learning and transportation resource constraints provided by the present invention on the four scale problems is better than these heuristic rules (the mean completion time is the smallest), which fully reflects the effectiveness of the method.

[0106] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A flexible workshop joint scheduling method based on graph neural network and reinforcement learning, characterized in that: The specific process is: Constructing a scheduling agent model, wherein the scheduling agent model includes a feature extraction network, an actor network, and a critic network; the feature extraction network includes a graph neural network and a fully connected neural network; Training the scheduling agent model: input the state information of the previous moment into the feature extraction network to obtain the extracted environmental features; input the environmental features into the Actor network to generate the action strategy of the agent at the current step, and generate the scheduling action based on the action strategy sampling; execute the scheduling action in the current environment to obtain the state at the current moment and the reward for executing this action, and calculate the reward value fed back by the environment to the agent based on the state; input the global features into the fully connected Critic network to calculate the state value; calculate the loss function based on the state value and the reward value, and update the parameters of the agent model based on the loss function; Based on the feature extraction network and actor network of the trained agent model, flexible joint scheduling of workshops is realized, and joint scheduling actions of processes, machines and unmanned vehicles are output; The specific process of this method is: S1, build the scheduling agent model including feature extraction network, Actor network and Critic network, and adjust the feature extraction network parameters , Actor network parameters and Critic network parameters Initialize; obtain flexible workshop environment data of different scales as training sets and test sets; S2, set the upper limit of training times to , if the number of training times reaches the preset value , then the training ends, otherwise, each flexible workshop environment is initialized: The data of each environment is converted into the original features of the workshop environment, and the initial input state information of the scheduling agent model is obtained. , ; S3: Let the scheduling agent model interact with each environment in parallel. Repeat the following steps for each environment times, the For the Number of processes in an environment: First, the state information of the previous moment Input to the feature extraction network to obtain the extracted environmental features; Secondly, the environmental features are input into the Actor network to generate the action strategy of the current agent , and the action strategy Sampling generates scheduling actions ; Then, execute the scheduling action in the current environment , get the current state and the reward for performing this action ; Finally, the global environment features are input into the Critic network to generate the state value , based on the state value, further calculate the advantage function value of the corresponding step , save the state, action, reward and advantage function to the experience buffer; S4, based on the data stored in the experience buffer, calculate the loss function of the PPO algorithm, and use the loss function to update the feature extraction network parameters , Actor network parameters and Critic network parameters ; S5, determine the current number of training times: If the current number of training times is The currently trained scheduling agent model is verified on the given validation set. If the average completion time of the currently generated scheduling action is less than the previous minimum completion time, the parameters of the current network are saved. If the current training times are If it is an integer multiple of , the environment set for training is regenerated; S6, repeatedly perform S2 to S5 until the number of training times reaches the preset value , and finally obtain the model parameters optimized for the current scale problem , , , that is, the optimized scheduling agent model is obtained, and the optimized scheduling agent model is used to realize flexible workshop joint scheduling.

2. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The flexible workshop environment data includes the processing time of the process on different machines and the transportation time between machines.

3. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The environmental features include: original features in the state information and features extracted by a feature extraction network, specifically: extracted process features and machine features, original unmanned vehicle features in the state information, and process-machine-unmanned vehicle combination features.

4. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 3 is characterized in that: The graph neural network is an edge feature graph attention network GAL, and the process of acquiring the environmental features is as follows: First, the original process features are input into the fully connected neural network MLP to obtain the extracted process features; Secondly, the process features are summed up according to the graph relationship as the edge features of the graph, and are input into the edge feature graph attention network GAL together with the original machine features to obtain the process features and machine features; Finally, the extracted process features, machine features, original unmanned vehicle features, and process-machine-unmanned vehicle triple combination features are concatenated as environmental features and used as input features for the subsequent Actor network.

5. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The reward function is: ， in, It is a process exist An estimate of the lower bound on the makespan for the time step.

6. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 5 is characterized in that: The advantage function value for: in, and is the set hyperparameter, is the number of processes to be performed, It is the state value output by the Critic network.

7. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 6 is characterized in that: The loss function is: is the mean square error between the estimated state value and the cumulative reward; Status The entropy of the strategy, expect; , is a positive coefficient, and are the actual update strategy and the exploration strategy respectively. is the cutoff coefficient; is the advantage function value.

8. The flexible workshop joint scheduling method based on graph neural network and reinforcement learning according to claim 7 is characterized in that: Update the feature extraction network parameters using the loss function , Actor network parameters and Critic network parameters for: in, , is the learning rate for gradient descent.

9. A flexible workshop joint scheduling device based on graph neural network and reinforcement learning, the device is used to implement the method as claimed in claim 1, characterized in that: The device is a scheduling agent model, including a feature extraction network and an Actor network; The feature extraction network is used to extract features from the state information to obtain extracted environmental features, including a graph neural network and a fully connected neural network; The fully connected neural network is used to receive the original process features and perform feature extraction to obtain process features, and add the process features according to the graph relationship as the edge features of the graph; The graph neural network is used to extract features from the edge features of the graph and the original features of the machine to obtain process features and machine features; The extracted process features, machine features, original unmanned vehicle features, and process-machine-unmanned vehicle triple combination features are concatenated as environmental features; The Actor network is used to generate the action strategy of the current step agent based on the input environmental characteristics and generate scheduling actions for the samples.

Images