Multi-action reinforcement learning method and system for flexible job shop scheduling
Through the deep learning DRL framework of the multi-pointer graph neural network model, the complexity of resource transfer constraints in flexible work workshop scheduling is solved, efficient scheduling decisions are realized, and the solution efficiency and performance of flexible work workshop scheduling problems are significantly improved.
Patent Information
- Application Number
- CN202411057554.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-06-05
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-19
AI Technical Summary
The actual production constraint of resource transfer is rarely considered in the existing flexible operation workshop scheduling problems, which makes it difficult for the algorithm to be directly used for scheduling problems with different features. The existing metaheuristic methods are inefficient and difficult to solve complex FJSP-RT scheduling. question.
The deep learning DRL framework of multi-pointer graph neural network model is adopted to extract features through heterogeneous graph representation and multi-model network, generate the probability distribution of actions, optimize the flexible work workshop scheduling model, and realize the scheduling plan that minimizes the maximum completion time.
It significantly improves the solution efficiency of FJSP-RT scheduling problems, provides efficient scheduling decisions, shows excellent generalization capabilities and performance, and is better than baseline scheduling rules.
Smart Images

Figure CN118982191A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of workshop scheduling, and in particular relates to a multi-action reinforcement learning method and system for flexible job shop scheduling. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.
[0003] Smart manufacturing aims to make production systems more intelligent and flexible by integrating cutting-edge technologies, and to provide efficient and customized production solutions for the manufacturing industry. Its core tasks involve using intelligent scheduling algorithms and systems to intelligently allocate manufacturing tasks (jobs) to appropriate manufacturing resources (machines), ultimately achieving optimal production efficiency and system performance.
[0004] As an extension of JSP, FJSP has extremely high research value in the context of intelligent manufacturing. In the actual Flexible Job-shop Scheduling Problem (FJSP) manufacturing environment, dynamic resource transfer between machines is crucial to adapt to production changes and optimize scheduling decisions. The inventors found that currently few flexible job-shop scheduling problems consider resource transfer (RT), an actual production constraint.
[0005] Compared with the general job shop scheduling problem, FJSP-RT has more complex features and has been proven to be strongly NP-hard. To address this challenge, academia usually uses heuristic and meta-heuristic methods to obtain feasible solutions in a reasonable time. Notable heuristic methods include priority scheduling rules, which iteratively assign jobs to machines by constructing an ordered scheduling method. On the other hand, meta-heuristic methods, such as evolutionary algorithms, explore the solution space through a complex search process to seek high-quality solutions. Overall, current research focuses on finding efficient and high-quality solutions to meet the complexity of FJSP and actual production needs.
[0006] The inventors found that the existing meta-heuristic methods have the following problems in solving the FJSP-RT scheduling problem: (1) Knowledge construction and transfer are difficult, which makes it difficult for the algorithm to be directly used to solve scheduling problems with different characteristics. (2) When solving scheduling problems, the algorithm often starts from scratch, so the algorithm efficiency is relatively low. Summary of the invention
[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multi-action reinforcement learning method and system for flexible job shop scheduling. Taking into account the complexity and uncertainty of the problem, compared with existing scheduling rules, metaheuristic algorithms and other algorithms, the proposed deep learning DRL framework based on multi-Pointer Graph Neural Network (multi-PGN) model has significant advantages in solving the FJSP-RT problem.
[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0009] In a first aspect, the present invention provides a multi-action reinforcement learning method for flexible job shop scheduling, comprising:
[0010] Obtain shop floor data for flexible job shop scheduling with resource transfer;
[0011] Establish a flexible job shop scheduling model with resource transfer based on shop floor data;
[0012] The flexible job shop scheduling model is optimized and solved by using a deep learning framework based on a multi-pointer graph neural network model to obtain the optimal production plan for the shop, and the workpieces in the shop are scheduled based on the optimal production plan for the shop; specifically, the following steps are included:
[0013] A heterogeneous graph is used to represent the real-time state of the flexible job shop scheduling model to obtain a state heterogeneous graph;
[0014] Inputting the state heterogeneous graph into a heterogeneous graph multi-model network to extract features, and obtaining a feature embedding representation; the feature embedding representation includes a process node embedding representation and a machine node embedding representation;
[0015] The feature embedding representation is input into a multi-pointer graph neural network model to generate a probability distribution of actions, thereby determining the actions of the scheduling plan.
[0016] According to a further technical solution, the optimization goal of the flexible job shop scheduling model is to minimize the maximum completion time.
[0017] According to a further technical solution, the real-time status includes three groups of features, namely, machine node, process node and OM arc features.
[0018] According to a further technical solution, the heterogeneous graph multi-model network includes a process node embedding process and a machine node embedding process.
[0019] A further technical solution is that the multi-pointer graph neural network model includes an operation strategy network and a machine strategy network, which respectively select operation actions and machine actions according to feature embedding representation, and the operation actions and machine actions together constitute an action.
[0020] According to a further technical solution, the operation strategy network and the machine strategy network have the same network structure but do not share parameters.
[0021] A further technical solution is to use a multi-proximal strategy optimization algorithm to train the multi-pointer graph neural network model.
[0022] In a second aspect, the present invention provides a multi-action reinforcement learning system for flexible job shop scheduling, comprising:
[0023] An acquisition module is configured to: acquire shop floor data of flexible job shop scheduling with resource transfer;
[0024] A scheduling model building module is configured to: build a flexible job shop scheduling model with resource transfer based on shop floor data;
[0025] The scheduling model scheduling module is configured to: optimize and solve the flexible job shop scheduling model using a deep learning framework based on a multi-pointer graph neural network model to obtain the optimal production plan for the workshop, and schedule the workpieces in the workshop based on the optimal production plan for the workshop; specifically, it includes:
[0026] A heterogeneous graph is used to represent the real-time state of the flexible job shop scheduling model to obtain a state heterogeneous graph;
[0027] Inputting the state heterogeneous graph into a heterogeneous graph multi-model network to extract features, and obtaining a feature embedding representation; the feature embedding representation includes a process node embedding representation and a machine node embedding representation;
[0028] The feature embedding representation is input into a multi-pointer graph neural network model to generate a probability distribution of actions, thereby determining the actions of the scheduling plan.
[0029] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a multi-action reinforcement learning method for flexible job shop scheduling as described in the first aspect.
[0030] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a multi-action reinforcement learning method for flexible job shop scheduling as described in the first aspect are implemented.
[0031] One or more of the above technical solutions have the following beneficial effects:
[0032] Taking into account the complexity and uncertainty of the problem, the proposed deep learning DRL framework based on a multi-Pointer Graph Neural Network (multi-PGN) model has significant advantages in solving the FJSP-RT problem compared with existing scheduling rules, metaheuristic algorithms and other algorithms.
[0033] For the first time, this paper designs a heterogeneous graph that efficiently expresses the state, which contains information about machines and operations, and embeds the state into it through a heterogeneous graph-based multi-model network (HGMN), a specially designed graph neural network (GNN), to achieve the goal of feature embedding, making it easy to obtain accurate and effective information about FJSP-RT from the heterogeneous graph.
[0034] This paper introduces a multi-action deep reinforcement learning (DRL) framework, describes a new definition of various elements (state, action, and reward) in the multi-decision process of solving FJSP, and improves the efficiency of solving the FJSP scheduling problem. A multi-pointer graph neural network model is developed to capture and analyze key information in heterogeneous graphs and promote accurate scheduling decisions.
[0035] By training and testing on randomly generated examples, the framework significantly outperforms the baseline scheduling rules at various scales and environments. In further experiments, the framework performs well in large-scale tests after small-scale training, showing excellent generalization ability. Overall, the method and system show excellent performance in solving scheduling problems. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0037] Figure 1 It is a FISP heterogeneous graph representation of an embodiment of the present invention;
[0038] Figure 2 A schematic diagram of a multi-model network based on a heterogeneous graph according to an embodiment of the present invention;
[0039] Figure 3 A multi-pointer graph neural network model diagram of an embodiment of the present invention;
[0040] Figure 4 This is a convergence curve diagram of the calculation example "10*5" of an embodiment of the present invention;
[0041] Figure 5 This is a convergence curve diagram of the calculation example "20*5" of an embodiment of the present invention;
[0042] Figure 6 This is a convergence curve diagram of the calculation example "10*10" of an embodiment of the present invention;
[0043] Figure 7 This is a convergence curve diagram of the calculation example "15*10" of an embodiment of the present invention;
[0044] Figure 8 This is a convergence curve diagram of the calculation example “20*10” of an embodiment of the present invention. DETAILED DESCRIPTION
[0045] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0047] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0048] The inventors found that few current flexible job shop scheduling problems consider resource transfer (RT), an actual production constraint. In this context, the present invention pays special attention to flexible job shop scheduling with resource transfer (FJSP-RT). At the same time, the introduction of resource transfer aims to more comprehensively consider the interaction between machines in the production process and how to more effectively coordinate these resources to improve the overall scheduling efficiency.
[0049] Embodiment 1
[0050] This embodiment provides a multi-action reinforcement learning method for flexible job shop scheduling, including the following steps:
[0051] Obtain shop floor data for flexible job shop scheduling with resource transfer;
[0052] Establish a flexible job shop scheduling model with resource transfer based on shop floor data;
[0053] The flexible job shop scheduling model is optimized and solved by using a deep learning framework based on a multi-pointer graph neural network model to obtain the optimal production plan for the shop, and the workpieces in the shop are scheduled based on the optimal production plan for the shop; specifically, the following steps are included:
[0054] A heterogeneous graph is used to represent the real-time state of the flexible job shop scheduling model to obtain a state heterogeneous graph;
[0055] Inputting the state heterogeneous graph into a heterogeneous graph multi-model network to extract features, and obtaining a feature embedding representation; the feature embedding representation includes a process node embedding representation and a machine node embedding representation;
[0056] The feature embedding representation is input into a multi-pointer graph neural network model to generate a probability distribution of actions, thereby determining the actions of the scheduling plan.
[0057] Among them, the workshop data of flexible job shop scheduling with resource transfer includes the number of workpieces, the number of processing machine tools, the processing time, the resource transfer rate, etc.
[0058] In this embodiment, the specific process of establishing a flexible job shop scheduling model with resource transfer based on shop data is as follows:
[0059] In the field of FJSP research, scholars were inspired by the FJCma notation introduced by Graham et al. In this notation, FJ stands for flexible job shop and Cma stands for maximum completion time (completion time). The mathematical model of FJSP is formulated using the notation in the following table.
[0060]
[0061] This embodiment extends the problem by introducing the setting of resource transfer (RT), namely FJSP-RT. This embodiment stipulates that the job must be inspected by a quality control inspector (QC) after completion. The number of QCs in each FJSP instance is fixed, and the amount of time these QCs take to inspect the job is different. The specific details are as follows: When the last operation of the job is completed, it means that the job has been processed. At this point, the job enters the quality inspection stage and is scheduled according to whether the QC is occupied and the priority. Each QC contains three pieces of information: priority, processing time, and the time when the last quality inspection was completed. The mathematical model of RT is represented using the symbols in the table below.
[0062]
[0063]
[0064] According to the relevant information of the job shop, a flexible job shop scheduling model with resource transfer is established, which includes:
[0065] (1) Assign each process to a machine tool (RS-1);
[0066] (2) Determine the resource transfer path (RS-2);
[0067] (3) Arrange and schedule the assigned processes on each machine tool (RS-3).
[0068] Based on the above description of FJSP-RT parameters and variables, the mathematical model constraints of FJSP-RT can be expressed as constraints (1)-(8).
[0069]
[0070] Target: C max (9)
[0071] In the mathematical model of FJSP-RT, the maximum completion time is defined as the maximum value of the final completion time of each job, and the optimization goal is to minimize the completion time (makespan).
[0072] In this embodiment, a deep learning framework based on a multi-pointer graph neural network model is used to optimize and solve the flexible job shop scheduling model to obtain the optimal production plan for the workshop. The specific process is as follows:
[0073] This embodiment regards the FJSP-RT problem as a multi-action reinforcement learning task, which is a multi-Markov decision process. In each round of iteration, a heterogeneous graph representation method is first used to clearly describe the current state of the problem, and then a heterogeneous graph multi-model network (HGMN) is used to extract relevant features from the operations and machine nodes in the graph to achieve the goal of feature embedding. Using these feature embeddings, the agent in the multi-pointer graph neural network model (multi-PGN) generates the probability distribution of actions and determines the planned actions by sampling. Finally, the actor-critic based multiple proximal policy optimization (multi-PPO) method is used to train the proposed multi-PGN model.
[0074] (1) State: The real-time state of FJSP-RT is represented by a heterogeneous graph, which can provide a more comprehensive and in-depth perspective. At time t, the scheduled and unscheduled states of all operations and machines in the problem constitute the current overall state st , denoted as H t =(O,M,C,E t ). From the description, we can know that the state s t Contains three groups of features: machine node, process node and OM arc features.
[0075] (2) Action: At time t, the agent performs a sequential decision process. At time t, the operation a o ∈A o (optional operation) and machine action a m ∈A m (compatible machine), the two together constitute action a=(a o ,a m ). The operation to be scheduled should meet the condition that the previous operation has been processed and the current operation has not been scheduled. The compatible machine to be scheduled should be idle.
[0076] (3) State transition function: At time t, the agent selects action a o (O ij ) and compatible machine actions a m (M k ) Execute the scheduled actions for this step. Determine an action (O ij ,M k ) and then retain the operation O ij and Machine M k The arc E ijk Delete O ij , update the features of the process nodes and machine nodes generated by the action, and advance the time step t to t+1 to obtain the new state H t+1 .
[0077] (4) Reward: This embodiment takes minimizing makespan as the optimization goal, which is reflected in the environment as minimizing C max It is worth noting that for a particular environment, there is only one unique C max value, and C after the last step of scheduling max is the makespan of the scheduling plan. The reward function is defined as follows:
[0078] r(s t ,a t ,s t+1 )=C max (s t )-C max (s t+1 ) (10)
[0079] It means that when the environment state is s at the scheduling time t t , select action at The agent gets the reward when s is reached, and the state is transferred to s t+1 .
[0080] (5) Strategy: Strategy π proposed in this embodiment θ (a o ,a m |s) consists of two random sub-strategies π θo (a o |s) and π θm (a m |s,a o ), where θ o and θ m is the parameter set of the operation selection strategy and the machine selection strategy. Θ=[θ o ,θ m ] is the parameter set for the entire strategy.
[0081] The above contents are described in detail below.
[0082] The feature extraction architecture established in this embodiment: Heterogeneous Graph Multi-Model Network HGMN: In order to obtain accurate and effective information about FJSP-RT from the heterogeneous graph, this embodiment proposes a HGMN (Heterogeneous Graph Multi-Model Network) customized for this problem, such as Figure 2 shown.
[0083] The heterogeneous graph multi-model network includes a process node embedding process and a machine node embedding process, which can also be called a word embedding strategy including process node word embedding and machine tool allocation word embedding.
[0084] Process node embedding method: Figure 2 It can be seen that the established process node embedding process goes through three stages: information transmission, information aggregation and feature fusion.
[0085] Information transmission mechanism: A process node has fewer neighbor nodes, but has various contact types. ij , its neighbor node L t (O ij ) includes the previous process node O i,j-1 , the next process node O i,j+1 、Machine node M k , and can be considered to include the node itself. Therefore, multiple different MLPs are directly used in process node embedding to deal with various contact types. From the above analysis, it can be seen that process node embedding requires five different MLPs, labeled MLP ξ1 ~MLP ξ5 It has L h Layer, dh dimensional hidden layer and d-dimensional output. Therefore, the process of embedding process nodes to pass information from operation and machine nodes to process nodes can be expressed as Equation 11, where is the machine node embedding vector.
[0086]
[0087] Information aggregation mechanism: Due to the process node O ij There are multiple types of contacts between adjacent nodes, which requires the multi-action reinforcement learning framework to emphasize local information. To this end, RGCN (Relational Graph Convolutional Network) is used to capture the complex relationships and mutual influences between operations. The specific calculation is shown in Equation 12:
[0088]
[0089] Process Node O ij There is a previous process node O i,j-1 and the next process node O i,j+1 Therefore, let r = 2 edge types, and the weight matrix is expressed as (r∈[0,1]). It has L h layers, h-dimensional hidden layers, and d-dimensional outputs.
[0090] Feature fusion method: Taking parameter ρ∈[0,1] as the ratio, the feature fusion is performed on the above process node embedding, as shown in Formula 13. This multi-level information fusion helps to improve the model's understanding of the process nodes.
[0091]
[0092] Machine node embedding method: Figure 2 It can be seen that the machine node embedding process goes through three stages: information transfer, self-enhancement, and feature fusion.
[0093] Information transfer GAT (Graph Attention Network) mechanism: A machine node has a large number of adjacent process nodes, and can focus on the most relevant part of information transfer. Therefore, this embodiment uses an improved version of GAT calculation as the first stage of machine node embedding. The details are as follows.
[0094] Step 1: Attention coefficients of adjacent process nodes.
[0095] First, at the scheduling time t, the process node O ij The corresponding radian E ijk Connection (feature vector concatenation), regarded as "process node O ij Along the arc Eijk To machine node M k Transmit information", that is, α ijk =[α ij ||γ ijk ]∈R 7 Secondly, due to the process node O ij When information is transmitted to adjacent machine nodes in different directions, the connected feature vectors are also different. Therefore, we define W M ∈R d ×3 and W O ∈R d×7 , respectively for machine node feature vector β k and the connected process node feature vector α ijk The linear mapping of d is the dimension of the embedded feature vector. In this embodiment, the dimension of the feature vector embedded in the machine and process nodes is the same. The attention coefficient of each adjacent process node to the machine node is:
[0096] e ijk =LeakyReLU(attn T [W M β k ||W O α ijk ]) (14)
[0097] Step 2: The attention coefficient of the machine node itself.
[0098] Since there is no arc information on the self-loop of the machine node, the machine node M k The importance to itself can be evaluated using Formula 15.
[0099] e kk =LeakyReLU(attn T [W M β k ||W M β k ]) (15)
[0100] So far, we have obtained the machine node M k The calculation method of the importance coefficients of all adjacent nodes along the connecting arc (including self-loops). Next, all attention coefficients are normalized to obtain the normalized attention coefficient λ ijk and λ kk , which is convenient for comparison and calculation.
[0101] Step 3: Weighted aggregation of information.
[0102] The calculation of machine node embedding needs to include processing information. t (Mk ) to machine node M k The information weighted aggregation process is defined as Equation 16, where is the machine node embedding vector.
[0103]
[0104] Self-enhancement: MLP mechanism: In order to improve the accuracy of machine actor selection, the original features of the machine nodes are independently input into the MLP to achieve self-enhancement, marked as MLP ξ0 It has L h Layer, d h dimensional hidden layer and d-dimensional output. The specific calculation is shown in Formula 17.
[0105]
[0106] Feature fusion mechanism: Finally, the obtained global information and local information are processed according to Formula 18. In this embodiment, a parameter ρ∈[0,1] is set as the fusion ratio.
[0107]
[0108] Message passing and stacking mechanism: By stacking L layers of HGMN, the effectiveness of feature extraction is significantly enhanced, and the final embedding representation is obtained: In addition, the embedding representation processed by the L-layer HGMN needs to be average pooled as follows:
[0109]
[0110] Through the above process, heterogeneous graphs of various sizes can be converted into embeddings of fixed dimensionality. The HGMN parameter set is denoted by η.
[0111] Multi-action framework multi-PGN model: As the last part of the deep learning framework based on the multi-pointer graph neural network model, two policy networks are designed to complete the decision-making part of the multi-action reinforcement learning framework. Specifically, at time t, the job actor and the machine actor select according to the embedded representation and Form a complete action The two actors have the same network structure but do not share parameters. See Figure 3 .
[0112] At this stage, the MLP layer is used to output the operation action coefficients and machine motion coefficient The calculation formula is as follows:
[0113]
[0114]
[0115] ·||· is a join operation. Operations that schedule or violate priority constraints can be and is set to -∞ to mask. Similarly, incompatible machines are masked as selected operations at time t. Then, the scores are normalized to probabilities using the Softmax function as follows:
[0116]
[0117] Based on and using different forecasting strategies and Such as sampling strategy, greedy strategy, etc., to select the most likely action and
[0118] In this embodiment, a multi-proximal strategy optimization algorithm is used to train the multi-pointer graph neural network model.
[0119] Multi-PPO: In multi-action decision-making tasks, the PPO algorithm is insufficient because it only contains one actor for learning a single policy and can only control one action per time step. To solve this problem, this example introduces multi-PPO, which extends the PPO algorithm from a single action to multiple actions and uses two actors to learn two sub-policies to make decisions in two action spaces.
[0120] Example analysis:
[0121] This embodiment uses training instances, test instances, and verification instances as evaluation instances. Instances for training and verification are randomly generated according to a given distribution method. Tests are performed on small and medium-sized instances and large-scale instances. It is worth noting that the FJSP-RT of this embodiment is derived from FJSP, so the instance used is combined with RT. The specific details are as follows.
[0122] The randomly generated examples in this embodiment are shown in Table 1. Six small and medium-sized problem instances (10×5, 20×5, 10×10, 15×10, 20×10, and 30×10) are selected to train and verify multi-PGN. At the same time, the trained model is directly applied to larger-scale problem instances (50×20 and 100×20) to test the generalization performance of the algorithm.
[0123] Table 1. Scale of examples generated by random generation method
[0124]
[0125] Experimental analysis: In this embodiment, the number of HGMN layers is set to 2 (L = 2), the dimension of the operation and machine node embedding vector is set to 8 (d = 8), and the number of MLP layers is set to 2 (L h =2), the dimension of each hidden layer is 128 (d h =128), the output dimension is 8 (d=8). In the actor network and the critic network (MLP θπo ,MLP θπm ), the number of MLP layers is set to 2 (L h =2), the dimension of each hidden layer is 128 (d h =128). In the multi-PPO algorithm, the hyperparameters c of the three loss functions χ ,c υ and c ω They are set to 0.2, 2, and 1 respectively. The optimizer uses the famous Adam optimizer and sets the learning rate to 1×10 -3 to optimize the proposed framework.
[0126] Solution quality analysis: The effectiveness and feasibility of the model are evaluated by training and testing on small and medium-sized instances (10×5, 20×5, 10×10, 15×10, 20×10, 30×10). Figure 4 The convergence curves of the results for each scale instance during training are shown. The results for each iteration represent the average total duration of the batch. According to the results in Table 2, the trained multi-PGN model outperforms all baseline scheduling rules in all scale instances. In addition, the performance advantage of the model over the baseline scheduling rules becomes more obvious as the problem size increases, while the performance of the baseline scheduling rules deteriorates as the problem size increases. Although the baseline scheduling rules slightly outperform the proposed method in computation time, this advantage is not significant compared to the significant improvement in solution quality.
[0127] Table 2 Experimental results analysis: Part I
[0128]
[0129]
[0130] Generalization Performance Analysis of the Method Established in the Present Invention: In addition, the applicability and transition capability of the proposed framework on various scale instances are evaluated. In this section, the trained multi-PGN models (20×10 and 30×10) are directly tested on large-scale instances (50×20 and 100×20). The experimental results are shown in Table 3, which show that in the test of large-scale instances, the models trained on 20×10 and 30×10 instances outperform the baseline scheduling rules at all scales.
[0131] Table 3 Experimental results analysis: Part II
[0132]
[0133]
[0134] The experimental results of the method established in this embodiment show that the proposed multi-PGN-based DRL framework has significant advantages in solving the FJSP-RT problem compared with existing scheduling rules, metaheuristic algorithms and other algorithms. By training and testing on randomly generated instances, the framework significantly outperforms the baseline scheduling rules at various scales and environments. In further experiments, the framework performs well in large-scale tests after small-scale training, showing excellent generalization ability. Overall, the framework shows excellent performance in solving scheduling problems.
[0135] Embodiment 2
[0136] This embodiment provides a multi-action reinforcement learning system for flexible job shop scheduling, including:
[0137] An acquisition module is configured to: acquire shop floor data of flexible job shop scheduling with resource transfer;
[0138] A scheduling model building module is configured to: build a flexible job shop scheduling model with resource transfer based on shop floor data;
[0139] The scheduling model scheduling module is configured to: optimize and solve the flexible job shop scheduling model using a deep learning framework based on a multi-pointer graph neural network model to obtain the optimal production plan for the workshop, and schedule the workpieces in the workshop based on the optimal production plan for the workshop; specifically, it includes:
[0140] A heterogeneous graph is used to represent the real-time state of the flexible job shop scheduling model to obtain a state heterogeneous graph;
[0141] Inputting the state heterogeneous graph into a heterogeneous graph multi-model network to extract features, and obtaining a feature embedding representation; the feature embedding representation includes a process node embedding representation and a machine node embedding representation;
[0142] The feature embedding representation is input into a multi-pointer graph neural network model to generate a probability distribution of actions, thereby determining the actions of the scheduling plan.
[0143] Embodiment 3
[0144] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of embodiment 1 when executing the program.
[0145] Embodiment 4
[0146] The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the method of embodiment 1 are performed.
[0147] The steps involved in the apparatus of the above embodiments 3 and 4 correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.
[0148] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
[0150] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A multi-action reinforcement learning method for flexible job shop scheduling, characterized in that: include: Obtain shop floor data for flexible job shop scheduling with resource transfer; Establish a flexible job shop scheduling model with resource transfer based on shop floor data; The flexible job shop scheduling model is optimized and solved by using a deep learning framework based on a multi-pointer graph neural network model to obtain the optimal production plan for the shop, and the workpieces in the shop are scheduled based on the optimal production plan for the shop; specifically, the following steps are included: A heterogeneous graph is used to represent the real-time state of the flexible job shop scheduling model to obtain a state heterogeneous graph; Inputting the state heterogeneous graph into a heterogeneous graph multi-model network to extract features, and obtaining a feature embedding representation; the feature embedding representation includes a process node embedding representation and a machine node embedding representation; The feature embedding representation is input into a multi-pointer graph neural network model to generate a probability distribution of actions, thereby determining the actions of the scheduling plan.
2. A multi-action reinforcement learning method for flexible job shop scheduling as claimed in claim 1, characterized in that: The optimization goal of the flexible job shop scheduling model is to minimize the maximum completion time.
3. A multi-action reinforcement learning method for flexible job shop scheduling as claimed in claim 1, characterized in that: The real-time status includes three groups of features, namely, machine node, process node and OM arc features.
4. A multi-action reinforcement learning method for flexible job shop scheduling as claimed in claim 1, characterized in that: The heterogeneous graph multi-model network includes a process node embedding process and a machine node embedding process.
5. A multi-action reinforcement learning method for flexible job shop scheduling as claimed in claim 1, characterized in that: The multi-pointer graph neural network model includes an operation strategy network and a machine strategy network, which respectively select operation actions and machine actions according to feature embedding representation, and the operation actions and machine actions together constitute an action.
6. A multi-action reinforcement learning method for flexible job shop scheduling as claimed in claim 5, characterized in that: The job strategy network and the machine strategy network have the same network structure but do not share parameters.
7. A multi-action reinforcement learning method for flexible job shop scheduling as claimed in claim 5, characterized in that: A multi-proximal strategy optimization algorithm is used to train the multi-pointer graph neural network model.
8. A multi-action reinforcement learning system for flexible job shop scheduling, characterized in that: include: An acquisition module is configured to: acquire shop floor data of flexible job shop scheduling with resource transfer; A scheduling model building module is configured to: build a flexible job shop scheduling model with resource transfer based on shop floor data; The scheduling model scheduling module is configured to: optimize and solve the flexible job shop scheduling model using a deep learning framework based on a multi-pointer graph neural network model to obtain the optimal production plan for the workshop, and schedule the workpieces in the workshop based on the optimal production plan for the workshop; specifically, it includes: A heterogeneous graph is used to represent the real-time state of the flexible job shop scheduling model to obtain a state heterogeneous graph; Inputting the state heterogeneous graph into a heterogeneous graph multi-model network to extract features, and obtaining a feature embedding representation; the feature embedding representation includes a process node embedding representation and a machine node embedding representation; The feature embedding representation is input into a multi-pointer graph neural network model to generate a probability distribution of actions, thereby determining the actions of the scheduling plan.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a multi-action reinforcement learning method for flexible job shop scheduling as described in any one of claims 1 to 7 are implemented.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the multi-action reinforcement learning method for flexible job shop scheduling according to any one of claims 1-7 are implemented.
Citation Information
Cited By
Double-resource flexible job shop dynamic scheduling method based on graph neural network
CN119886649A
Labor organization mode optimization method and system based on AI agent center
CN122288224A
