Flexible job shop scheduling method based on PAH-Attn-DRL

Abstract

The flexible job shop scheduling method based on PAH-Attn-DRL disclosed in the application models the scheduling process with flexible job shop scheduling commonality into a Markov decision process; constructs a parallel heterogeneous attention network and a deep reinforcement learning decision network based on an actor-critic architecture; trains the constructed deep neural network to obtain a flexible job shop scheduling method model combining the parallel heterogeneous attention network and the deep reinforcement learning; and solves the model. The application can improve the operation attention mechanism of the time sequence decoupling to model the time sequence constraints among operations explicitly, and the machine attention mechanism of the dynamic edge enhancement to adjust the modeling strength of the machine competition relationship adaptively, so that more information in small-scale data can be fully utilized to learn a scheduling strategy with more generalization ability, thereby achieving better results on large-scale problems.

Description

Technical Field

[0001] This invention belongs to the field of flexible job shop scheduling technology, specifically relating to a flexible job shop scheduling method based on PAH-Attn-DRL. Background Technology

[0002] Manufacturing is a crucial pillar of the national economy, and its core competitiveness largely depends on production efficiency and the level of intelligence. With the increasing demand for personalized products and the mainstreaming of multi-variety, small-batch production, traditional production scheduling systems are facing unprecedented challenges. Simultaneously, with the rapid development of technologies such as artificial intelligence, enterprises are moving towards intelligent and flexible manufacturing methods. The Flexible Job-shop Scheduling Problem (FJSP), as an important extension of the Job-shop Scheduling Problem (JSP), allows a single process to be selected for processing on multiple compatible machines, better simulating the real-world scenarios of modern flexible manufacturing systems. It has become one of the key research problems for improving production efficiency and reducing costs, and is a typical NP-hard problem. FJSP is widely used in manufacturing fields such as aircraft assembly, chemical production, and semiconductor manufacturing.

[0003] For a long time, researchers have been dedicated to developing efficient methods for solving FJSP problems. Traditional methods can be divided into exact methods, heuristic methods, and metaheuristic methods. Exact methods aim to find the global optimal solution to a problem through rigorous mathematical derivation, including branch and bound and integer programming. While exact methods guarantee the optimality of the solution, their computation time increases exponentially with the problem size. Heuristic methods are rules based on experience or common sense, the most typical being priority dispatch rules (PDRs). However, heuristic methods generally cannot guarantee the optimal solution, and their design is highly dependent on professional knowledge. Metaheuristic methods are algorithms designed based on inspiration from natural phenomena or biological laws, including genetic algorithms, simulated annealing algorithms, and particle swarm optimization algorithms, which can find high-quality solutions in a reasonable time. However, the performance of metaheuristic algorithms is highly dependent on parameter settings and operator selection, and usually requires careful design for specific problems. As a powerful technology in the field of artificial intelligence, deep reinforcement learning (DRL) has shown great potential in solving combinatorial optimization problems. Inspired by the application of DRL in combinatorial optimization problems such as the Traveling Salesman Problem and the Vehicle Routing Problem, many scholars have recently used DRL methods to solve the Functional Junction Problem (FJSP). DRL, with its powerful end-to-end learning and decision-making capabilities, provides a new paradigm for solving FJSP. DRL agents can learn approximate optimal scheduling strategies directly from high-dimensional raw data through continuous interaction with the environment, without complex human intervention, demonstrating great potential. Although DRL methods have achieved good results in solving FJSP, two main limitations remain: first, in terms of feature extraction, the modeling of the relationship between operations and the machine is still insufficient; second, the solution quality still needs improvement. Therefore, how to design more expressive feature extraction networks remains a pressing research area. Summary of the Invention

[0004] The purpose of this invention is to provide a flexible job shop scheduling method based on PAH-Attn-DRL, which aims to solve the problem of insufficient accuracy in operation and machine feature extraction of existing flexible job shop scheduling methods.

[0005] The technical solution adopted in this invention is a flexible job shop scheduling method based on PAH-Attn-DRL. First, the FJSP is modeled, defining elements such as state, action, and reward. For feature extraction, a PHGT-Net network is constructed, employing a parallel architecture of GAT and Transformer to extract local structural features and global dependency features respectively. An improved temporally decoupled operation attention mechanism explicitly models temporal constraints between operations, and a dynamically edge-enhanced machine attention mechanism adaptively adjusts the modeling intensity of machine competition relationships. Local and global features are fused through cross-attention, and a decision network is constructed based on an actor-judge architecture, trained using the PPO-Clip algorithm. Specifically: Step 1: Model the scheduling process, which shares common characteristics with flexible job shop scheduling, as a Markov decision process; Step 2: Construct a parallel heterogeneous attention network under flexible job shop scheduling to extract operational and machine features; Step 3: Construct a deep reinforcement learning decision network based on an actor-judge architecture; Step 4: Train the deep neural network consisting of a scheduling method architecture that combines parallel heterogeneous attention networks and deep reinforcement learning strategies to obtain a flexible job shop scheduling method model. Step 5: Solve the scheduling scenarios that share common characteristics with flexible job shop scheduling in the flexible job shop scheduling method model.

[0006] The invention is further characterized in that, Step 1 is as follows: Step 1.1: Construct a state space based on feature vectors of operation and machine information. ,operate and machines The features of each type contain 8 types of information, while the features of an operation-machine pair contain 6 types of information. Each operation Features include: 1) Scheduling flag: 1 for scheduled tasks, 0 for unscheduled tasks; 2) Minimum processing time on compatible machines; 3) Average processing time on compatible machines; 4) Remaining processing time; 5) Waiting time, currently unschedulable operations are recorded as 0; 6) Processable Operations The percentage of machines; 7) Related Tasks Unscheduled operands in; 8) Related tasks The remaining processing time, operation The average processing time for unscheduled operations.

[0007] Each machine Features include: 1) Free space flag: 0 for free space, 1 for not free space; 2) Compatible unscheduled operands; 3) The currently available candidate operands; 4) The shortest processing time for all compatible unscheduled operations; 5) Average processing time for all compatible unscheduled operations; 6) Expected idle time; 7) Waiting time; 8) Utilization rate, the ratio of non-idle time to total production time; Each compatible operation—machine pair The features include the following information: 1) Processing time ; 2) Processing time and The ratio of maximum processing time; 3) Processing time and The ratio of the maximum processing time for compatible unscheduled operations; 4) Processing time The ratio of the maximum processing time to the maximum processing time of a compatible operating-machine pair; 5) Processing time and The ratio of the remaining processing time; 6) and The sum of waiting times.

[0008] Step 1.2: Constructing the basic motion space A single scheduling operation selects one operation and assigns it to a compatible machine; the operation selection and machine selection are considered as a whole, and the action is the time step. A feasible operation—machine to

[0009] Step 1.3: Construct the basic reward function r set up For state The goal of the flexible job shop scheduling problem is to minimize the maximum completion time (makespan) of the already scheduled operations. Therefore, a reward will be given. Defined as time step and The difference between the makespan of the obtained scheduling scheme, i.e. The goal of reinforcement learning is to learn a strategy. To maximize expected total return: When the discount factor At that time, the cumulative reward is Therefore, maximizing the cumulative reward is equivalent to minimizing the final makespan. , This represents the total time step; Step 1.4: Construct the basic state transition function Perform an action Then, the state transition function is defined as ; Specific update rules include: Distributed to machine Determined in the disjunction diagram and Prioritize other scheduled processes; update Load time and Completion time; Remove from the set to be scheduled Update the set of ready operations; if all operations have been scheduled, proceed to the terminated state; and update the disjunction graph, changing the state from [previous state]. Transition to .

[0010] Step 2 is as follows: Step 2.1: Construct a temporally decoupled operation attention mechanism. By introducing a trainable bias, the neighbors of the operation node are clearly distinguished as predecessors and successors, thereby achieving decoupling and explicit modeling of the temporal relationship between operations. Step 2.2: Construct a dynamic edge-enhanced machine attention mechanism. The machine node features are linearly transformed through a node linear layer, and the edge features are multiplied by an edge linear layer and an edge feature enhancement network to obtain edge-enhanced features. This enables the model to adaptively adjust the modeling intensity of machine competition relationships. Step 2.3: Construct a multi-head attention mechanism for inter-head interaction; Step 2.4: Construct a cross-attention mechanism to fully couple the different dimensional features captured by GAT and multi-head attention mechanisms; Step 2.5: Through linear layer adaptive fusion, operational features for enhanced fusion are formed, and pooling is performed on the features after cross-attention fusion to obtain auxiliary global features.

[0011] Step 2.1 is as follows: Through calculation operations Attention coefficients with neighboring nodes are used to obtain operational relevance and identify key operations. Considering that for each operation node, its neighboring nodes are divided into predecessor and successor nodes, for input features... operation nodes The attention coefficients between the node and its predecessor and successor nodes are calculated as follows:

[0012] in, and Represents a linear transformation; Indicates the activation function; For bias, as shown below:

[0013] in, , For the attention coefficient, the standardized attention coefficient is obtained by normalization using the Softmax function. :

[0014] Finally, a weighted linear combination of the features after linear transformation of neighboring nodes is performed, and a Sigmoid activation function is applied. Obtain the operational feature vector: .

[0015] Step 2.2 is as follows: In the production process, machines capable of handling the same operation are called competing machines. These competing machines compete for unscheduled operations that they are compatible with. The intensity of this competition varies among these machines. , indicating machine and machines The intensity of competition between them; based on the operational characteristic vectors of the competing machines. To determine the intensity of competition, a lightweight multilayer perceptron (MLP) is introduced to map the original edge features to dynamic weights of 0-1, realizing a machine attention mechanism for dynamic edge enhancement; specifically, for the machine... The original competition vector First, the weights are generated by passing through two layers of feedforward network and using Sigmoid activation:

[0016] in, The Sigmoid activation ensures the output falls within the [0, 1] interval, acting as a gating mechanism; subsequently, dynamic weights are used to enhance edge features, achieving context-aware modulation.

[0017] in, It is a calibration The weight matrix is ​​then used, and finally, the modulated edge features are concatenated with the machine node features to obtain the inter-machine attention coefficients.

[0018] in, It is a calibration and The weight matrix, Represent a linear transformation; , They are machines and machines Node characteristics; Subsequently, by performing Softmax normalization, weighted combination, and activation on the attention coefficients, the final machine feature vector is obtained. :

[0019] in, Is with A set of machines that are in competition with each other; Step 2.3 is as follows: Construct a multi-head attention mechanism for inter-head interaction, with a set number of attention heads. First use The attention heads with independent parameters are computed in parallel to obtain... Equal-dimensional features Then, following the aggregation strategy of the GAT architecture, except for the last layer which uses an averaging operator, the remaining layers are spliced ​​together, and the spliced ​​or averaged results are then processed. Send to the head-to-head interaction layer to generate for each location Dynamic weights Next, the multi-head results are weighted and summed using dynamic weights to obtain the weighted fused features; finally, an activation function is applied. Output the final weighted fused features: . Steps 2.4 and 2.5 are as follows: Step 2.4: Construct a cross-attention mechanism to fully couple the different dimensional features captured by GAT and multi-head attention mechanisms; [The text abruptly ends here, likely due to an incomplete sentence or missing information.] The outputs after the layer network are denoted as follows: Using local features of the GAT as queries and global features of the Transformer as keys and values, this approach maintains the dominance of local priority information in the current scheduling state while ensuring accurate injection of global information. The features fused through cross-attention are denoted as... and ; Step 2.5: Stitching and Pooling Through linear layer adaptive fusion, operational features for enhanced fusion are formed. and machine characteristics ; Furthermore, pooling is performed on the features after cross-attention fusion to obtain auxiliary global features:

[0020] in, , These represent the sets of operations and machines, respectively.

[0021] Step 3 is as follows: Step 3.1: Building an actor network Its neural network parameters are First, the extracted operational features, machine features, auxiliary global features, and the calculated operational-machine pair features are concatenated and input into the actor network to generate each action in the following manner. Corresponding scalar :

[0022] Then, the probability distribution for selecting each action is output using the Softmax function:

[0023] Step 3.2: Building a network of judges Using pooled auxiliary global features As input, a scalar is generated as an estimate of the state value.

[0024] Step 4 specifically includes: Step 4.1: Initialize hyperparameters Set the experience replay pool size and the minimum batch size. b Set the neural network learning rate, set the hidden layer dimension, and set the discount rate. The number of GAT and Transformer layers in both the operational and machine feature extraction networks is set to the same number. Both the actor network and the judge network have two MLP layers, and GAE parameters are set accordingly. and discount factor parameter; Step 4.2: Based on the state dimension in the flexible workshop scheduling environment, set the input dimension of the actor network in Step 3.1, set the output dimension based on the number of machine pairs in the environment, set the activation function to ReLU, and perform SoftMax normalization on the output results; based on the observation dimension, set the input dimension of the judge network in Step 3.2, set the network output dimension to 1, and set the activation function to ReLU. Step 4.3: Receive the state given by the flexible workshop scheduling environment at the start of each round, as input to the actor network and the judge network. The actor network outputs the action probability value of the agent in the environment. If the action is not feasible, its probability is set to 0. The judge network outputs the estimated state value corresponding to the action. Here, the agent refers to the scheduling decision-maker / scheduler itself, not the physical entity in the workshop (such as a machine or workpiece). Step 4.4: After each time step in the flexible workshop scheduling scenario, collect the current state information, actions, rewards, and the next state information. Stored in the experience replay pool, awaiting training; Step 4.5: During training, a minimum batch is extracted from the data in the experience replay pool, and the network is updated using the Adam optimizer to minimize the composite loss, which includes policy gradient pruning, value function, and entropy regularization. After training, a flexible job shop scheduling method model based on PAH-Attn-DRL is obtained.

[0025] Compared with the prior art, the present invention has the following beneficial effects: This invention innovatively constructs a deep fusion framework of parallel heterogeneous attention networks and deep reinforcement learning, breaking through the technical limitations of traditional FJSP solution methods in terms of single feature extraction dimension, implicit temporal constraint modeling, and static characterization of machine competition relationships. Through the parallel collaborative architecture of GAT and Transformer, it achieves synchronous capture and organic fusion of local topology and global long-range dependencies, significantly improving the state representation capability and decision information completeness in complex scheduling scenarios. The improved temporal decoupling operation attention mechanism is the first to explicitly and accurately model the pre- and post-order constraints between operations, effectively alleviating the scheduling scheme infeasibility problem caused by fuzzy temporal relationship encoding in traditional methods. The dynamic edge-enhanced machine attention mechanism endows the system with adaptive perception and dynamic response capabilities to machine competition situations, realizing real-time adaptation and flexible optimization of scheduling strategies to production line load fluctuations. Based on the actor-judge architecture and the stable training paradigm of the PPO-Clip algorithm, it further ensures the convergence efficiency and solution quality stability of strategy optimization. Overall, this invention has achieved significant improvements in core indicators such as solution efficiency, feasibility, dynamic adaptability, and scalability. It provides a technical paradigm for intelligent scheduling decision-making in large-scale complex manufacturing environments that combines theoretical foresight with industrial application value, and has important demonstrative significance for promoting autonomous and controllable optimization decision-making in the field of intelligent manufacturing. Attached Figure Description

[0026] Figure 1 This is a flowchart of the flexible job shop scheduling method based on PAH-Attn-DRL of the present invention; Figure 2 This is a structural diagram of the flexible job shop scheduling method based on PAH-Attn-DRL of the present invention; Figure 3 This is a diagram of the parallel heterogeneous attention network structure in the flexible job shop scheduling method based on PAH-Attn-DRL of this invention; Figure 4 This is a diagram of the temporally decoupled operation attention network structure in the flexible job shop scheduling method based on PAH-Attn-DRL of this invention; Figure 5 This is a diagram of the dynamically edge-enhanced machine attention structure in the flexible job shop scheduling method based on PAH-Attn-DRL of this invention; Figure 6 This is a diagram of the deep reinforcement learning decision network structure based on the actor-judge architecture in the flexible job shop scheduling method based on PAH-Attn-DRL of this invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0028] Example 1 This invention relates to a flexible job shop scheduling method based on PAH-Attn-DRL, such as... Figure 1-6 As shown, the details are as follows: The flexible job shop scheduling problem used in this embodiment is a standard multi-variety, small-batch discrete manufacturing scenario, which includes 4 jobs and 5 machines. Each job process has machine flexibility and sequence constraints to verify the effectiveness of the present invention under the objective of minimizing the maximum completion time.

[0029] 1. Description of general scenarios for flexible workshop scheduling A standard flexible job shop scheduling problem can be formulated as follows. Given a set of... A collection of tasks and A collection of machines , recorded as Each assignment All are subject to priority constraints. A series of operations Composition. Homework Operation Can be used on qualified machines Processing is performed on top. Operation In the machine The processing time on is expressed as Assuming at the moment the system begins production... At this point, all jobs and machines are ready, and each operation, once started, cannot be interrupted, with each machine handling only one operation at a time. FJSP's task is to assign each operation to a compatible machine and determine the processing order of all operations on each machine. The optimization objective is to minimize the longest completion time of all jobs. ,in For operation The completion time.

[0030] For example, a manufacturing workshop needs to process N different types of workpieces. Each workpiece has a specific process route (process sequence), and each process can be processed on multiple candidate machines. The efficiency and cost of processing the same process vary among different machines. The scheduling objective is to minimize the maximum completion time (makespan) and the total machine load. The following will describe a scheduling scenario with common characteristics of flexible job shop scheduling. 2. Scheduling scenarios with common characteristics of flexible workshop scheduling A precision machining plant undertakes small-batch processing of various types of parts. The workshop contains... Taiwanese CNC machine tool (Including 3 vertical machining centers and 2 CNC lathes) The machine to be processed Typical parts ( This includes shafts, discs, housings, and plates. Each type of part has a fixed process route, and some processes can be performed on multiple machine tools, but the processing efficiency varies. The workshop operates on a two-shift system, aiming to complete daily orders without overtime. This scenario can be directly applied to industries such as mold manufacturing, aerospace component processing, and medical device manufacturing. The five machines are assembled as shown in the table below:

[0031] The four types of assignments are shown in the table below:

[0032] Process and processing time matrix as follows: 1) Homework Shaft parts

[0033] 2) Job J2 (Disc-type parts)

[0034] 3) Operation J3 (Shell Parts)

[0035] 4) Job J4 (plate parts)

[0036] Example 2 Step 1: Model the scheduling process, which shares common characteristics with flexible job shop scheduling, as a Markov decision process. Step 1 specifically includes: Step 1.1: Construct the basic state space A set of feature vectors representing operation and machine information. Both operation and machine features are designed to contain 8 types of information, while the operation-machine pair features contain 6 types of information. Specifically: Each operation The features include the following information: 1) Scheduling flag (1 for scheduled, 0 for unscheduled); 2) Shortest processing time on compatible machines; 3) Average processing time on compatible machines; 4) Remaining processing time; 5) Waiting time (0 for currently unschedulable operations); 6) Processable... 7) The percentage of machines; Unscheduled operands in the sequence; 8) The job to which it belongs. Remaining processing time (job) The average processing time for unscheduled operations.

[0037] Each machine The features include the following information: 1) Idle flag (0 for idle, 1 for non-idle); 2) Number of compatible unscheduled operations; 3) Number of currently available candidate operations; 4) Shortest processing time for all compatible unscheduled operations; 5) Average processing time for all compatible unscheduled operations; 6) Expected idle time; 7) Waiting time; 8) Utilization (ratio of non-idle time to total production time).

[0038] Each compatible operation—machine pair The features include the following information: 1) Processing time ;2) and The ratio of maximum processing time; 3) and The ratio of the maximum processing time of compatible unscheduled operations; 4) The ratio of the maximum processing time to the maximum processing time of a compatible operation-machine pair; 5) and The ratio of the remaining processing time; 6) and The sum of waiting times.

[0039] Step 1.2: Constructing the basic motion space In this invention, a single scheduling operation selects one operation and assigns it to a compatible machine. The invention treats these two steps—operation selection and machine selection—as a single unit. The action is defined as a time step. A feasible operation—machine to

[0040] Step 1.3: Construct the basic reward function The goal of FJSP is to minimize the maximum completion time (makespan), therefore, a reward will be given. Defined as time step and The difference between the makespan of the obtained scheduling scheme, i.e. When the discount factor At that time, the cumulative reward is Therefore, maximizing the cumulative reward is equivalent to minimizing the makespan.

[0041] Step 1.4: Construct the basic state transition function Perform an action Afterwards, the environment (the state of all operations and machines) will change and the disjunction graph will be updated, with the state changing from... Transition to .

[0042] Step 2: Construct the parallel heterogeneous attention network PHGT-Net under this flexible job shop scheduling to extract operational features and machine features; Step 3: Construct a deep reinforcement learning decision network based on an actor-judge architecture; Step 4: Train the deep neural network constructed by the PAH-Attn-DRL-based flexible job shop scheduling method architecture obtained in Steps 1 to 3 to obtain the PAH-Attn-DRL-based flexible job shop scheduling method model. Step 5: Solve the scheduling scenarios that share common characteristics with flexible job shop scheduling in the flexible job shop scheduling method model.

[0043] Example 3 Based on Example 2, step 2 specifically includes: Step 2.1: Construction of a temporally decoupled operation attention mechanism. By introducing a trainable bias, the neighbors of the operation node are clearly distinguished as predecessors and successors, thereby achieving decoupling and explicit modeling of the temporal relationship between operations.

[0044] Through calculation operations Attention coefficients with neighboring nodes can be used to obtain operational relevance and identify key operations. Considering that for each operational node, its neighboring nodes can be divided into predecessor and successor nodes, they can be temporally decoupled. By refining the neighbor relationship into predecessor and successor considerations, and introducing predecessor and successor biases, the features of the predecessor, itself, and the successor are decoupled, allowing for separate attention. Its predecessor and successor The relationship between these parameters is used to construct a temporally decoupled operational attention mechanism. The bias is a trainable parameter introduced into the feature matrix during forward propagation via addition. It distinguishes the temporal semantics of predecessors and successors and is dynamically adjusted to adapt to specific tasks. Specifically, for input features... operation nodes The attention coefficients between the node and its predecessor and successor nodes are calculated as follows:

[0045] in and It is a linear transformation. . For bias, as shown below.

[0046]

[0047] Furthermore, for the first and last operations, their predecessors or successors do not exist, so these illegal nodes are masked. Directly removing illegal nodes might lead to problems such as vanishing gradients and training instability. However, since a bias is incorporated before masking, the bias effect of illegal nodes can still be indirectly passed through the attention coefficients of legal nodes, preserving temporal semantic information. This allows the model to better understand the sequence structure of FJSP, while avoiding the vanishing gradient problem, thereby improving training stability and generalization ability. For the attention coefficients, a Softmax function is used to normalize them to obtain standardized attention coefficients. :

[0048] Finally, a weighted linear combination of the features after linear transformation of neighboring nodes is performed, and a Sigmoid activation function is applied. Obtain the operational feature vector:

[0049] The temporally decoupled operation attention mechanism enables each operation node feature to incorporate its temporal neighbor information, achieving structured perception and feature extraction of temporal constraints between operations.

[0050] Step 2.2: Constructing a dynamic edge-enhanced machine attention mechanism. The machine node features are linearly transformed through a node linear layer, and the edge features are multiplied by an edge linear layer and an edge feature enhancement network to obtain edge-enhanced features. This enables the model to adaptively adjust the modeling intensity of machine competition relationships.

[0051] Machines capable of handling the same operation are called competing machines. During production, competing machines compete for unscheduled operations that they are compatible with. The intensity of competition between competing machines varies. , indicating machine and machines The intensity of competition between them. Based on the operational feature vectors of the competing machines. The intensity of competition is determined by the number and importance of each operation; more operations mean greater competition. By introducing a lightweight multilayer perceptron (MLP) to map the original edge features to dynamic weights of 0-1, a dynamic edge-enhanced machine attention mechanism is implemented, enabling the model to better perceive the intensity of competition. Specifically, for machine attention... The original competition vector First, the weights are generated by passing through two layers of feedforward network and using Sigmoid activation:

[0052] in The Sigmoid function is activated to ensure the output falls within the [0, 1] interval, effectively acting as a gate. Then, dynamic weights are used to enhance edge features, achieving context-aware modulation.

[0053] in, It is a calibration The weight matrix is ​​then used, and finally, the modulated edge features are concatenated with the machine node features to obtain the inter-machine attention coefficients.

[0054] in, It is a calibration and The weight matrix, Represent a linear transformation; , They are machines and machines Node characteristics; Subsequently, by performing similar Softmax normalization, weighted combination, and activation on the attention coefficients, the final machine feature vector is obtained. :

[0055] in Is with A set of machines that are in competition with each other.

[0056] The dynamic edge-enhanced machine attention mechanism improves the model's ability to represent complex competitive relationships, providing richer machine state information for subsequent scheduling decisions.

[0057] Step 2.3: Construct a multi-head attention mechanism for inter-head interaction to overcome the redundancy or conflict problems caused by the independence and lack of coordination between heads in traditional multi-head attention.

[0058] Taking attention as an example, suppose the number of attention heads is... First use The attention heads with independent parameters are computed in parallel to obtain... Equal-dimensional features Then, following the aggregation strategy of the GAT architecture, except for the last layer which uses an averaging operator, the remaining layers are concatenated. The concatenated or averaged results are then... The data is fed into the inter-head interaction layer (which includes two feedforward layers and Softmax) to generate data for each location. Dynamic weights Next, the multi-head results are weighted and summed using dynamic weights to obtain the weighted fused features. Finally, an activation function is applied. Output the final result:

[0059] Step 2.4: Construct a cross-attention mechanism to fully couple the different dimensional features captured by GAT and Transformer.

[0060] Will pass The outputs after the layer network are denoted as follows: Using local features of the GAT as queries and global features of the Transformer as keys and values, this approach maintains the dominance of local priority information in the current scheduling state while precisely injecting global information. In this way, each node (operation or machine) can dynamically absorb global information while retaining its own local features, achieving stronger local-global feature fusion. The features fused through cross-attention are denoted as... and .

[0061] Step 2.5: Through linear layer adaptive fusion, operational features for enhanced fusion are formed, and pooling is performed on the features after cross-attention fusion to obtain auxiliary global features.

[0062] Example 4 Based on Example 3, step 2.5 specifically involves: Although the cross-attention mechanism has fused local and global features, it still retains GAT local features and concatenates them with the fused features to ensure that the local structural prior is not diluted. Furthermore, through adaptive fusion using linear layers, a dynamic balance between the local structural prior and global information is achieved. The resulting enhanced fused feature is denoted as... and .

[0063] Furthermore, the features fused through cross-attention are pooled to smooth the feature fusion, improve the generalization of the policy network, and complement the global features of the Transformer. This provides auxiliary global information to the decision network, significantly enhancing the downstream policy's ability to perceive the overall scale and distribution, thereby strengthening the overall representation ability and obtaining auxiliary global features.

[0064] in, , These represent the sets of operations and machines, respectively.

[0065] Example 5 Based on Example 4, step 3 is as follows: Step 3.1: Building an actor network Its neural network parameters are First, the extracted operational features, machine features, pooling auxiliary features, and the calculated operational-machine pair features are concatenated and input into the actor network to generate each action in the following manner. Corresponding scalar :

[0066] Then, the probability distribution for selecting each action is output using the Softmax function:

[0067] Step 3.2: Judge Network Construct, using pooled auxiliary global features As input, a scalar is generated as an estimate of the state value.

[0068] Example 6 Based on Example 5, step 4 specifically includes: Step 4.1: Initialize hyperparameters Set the experience replay pool size to Set minimum batch b The learning rate of the neural network is set to 4e-4, the hidden layer dimension is set to 64, and the discount rate is set to 64. The GAT and Transformer layers of both the operational and machine feature extraction networks are set to 2, and 4 attention heads are used for each. The actor network and judge network both have 2 MLP layers, and the GAE parameter is 0.99. and discount factor They were set to 0.2, 0.98, and 1 respectively.

[0069] Step 4.2: Based on the state dimension in the flexible workshop scheduling environment, set the input dimension of the actor network in Step 3.1, set the output dimension based on the number of machine pairs in the environment, and set the activation function to ReLU. Perform SoftMax normalization on the output results. Based on the observation dimension, set the input dimension of the judge network in Step 3.2, set the network output dimension to 1, and set the activation function to ReLU.

[0070] Step 4.3: Receive the state given by the flexible workshop scheduling environment at the start of each round, as input to the actor network and the judge network. The actor network outputs the action probability value of the agent in the environment. If the action is not feasible, its probability is set to 0. The judge network outputs the estimated state value corresponding to the action.

[0071] Step 4.4: After each time step in the flexible workshop scheduling scenario, collect the current state information, actions, rewards, and the next state information. The data is stored in the experience replay pool and awaits training.

[0072] Step 4.5: Perform parameter updates 4 times per epoch, extract the minimum batch based on the data from the empirical replay pool, update the network using the Adam optimizer, and minimize the composite loss that includes policy gradient clipping, value function and entropy regularization.

[0073] Step 4.6: After every 10 epochs, use a fixed validation dataset. Evaluate policy performance on data distributed in the same way as the training data. Resample the environment every 20 epochs to maintain the diversity of the training distribution.

[0074] Step 4.7: The total number of epochs for the multi-agent system reaches 1000, training is complete, and the model is saved as a flexible job shop scheduling method model based on PAH-Attn-DRL.

Claims

1. A flexible job shop scheduling method based on PAH-Attn-DRL, characterized in that, Specifically as follows: Step 1: Model the scheduling process, which shares common characteristics with flexible job shop scheduling, as a Markov decision process; Step 2: Construct a parallel heterogeneous attention network under flexible job shop scheduling to extract operational and machine features; Step 3: Construct a deep reinforcement learning decision network based on an actor-judge architecture; Step 4: Train the deep neural network consisting of a scheduling method architecture that combines parallel heterogeneous attention networks and deep reinforcement learning strategies to obtain a flexible job shop scheduling method model. Step 5: Solve the scheduling scenarios that share common characteristics with flexible job shop scheduling in the flexible job shop scheduling method model.

2. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 1, characterized in that, Step 1 is as follows: Step 1.1: Construct a state space based on feature vectors of operation and machine information. ,operate and machines The features of each type contain 8 types of information, while the features of an operation-machine pair contain 6 types of information. Step 1.2: Constructing the basic motion space A single scheduling operation selects one operation and assigns it to a compatible machine; the operation selection and machine selection are considered as a whole, and the action is the time step. A feasible operation—machine to ; Step 1.3: Construct the basic reward function set up For state The goal of the flexible job shop scheduling problem is to minimize the maximum completion time (makespan) of the already scheduled operations. Therefore, a reward will be given. Defined as time step and The difference between the makespan of the obtained scheduling scheme, i.e. ; The goal of reinforcement learning is to learn a strategy. To maximize expected total return: When the discount factor At that time, the cumulative reward is Therefore, maximizing the cumulative reward is equivalent to minimizing the final makespan. , This represents the total time step; Step 1.4: Construct the basic state transition function Perform an action Then, the state transition function is defined as ; The specific update rules include: Distributed to machine Determined in the disjunction diagram and Prioritize other scheduled processes; update Load time and Completion time; Remove from the set to be scheduled Update the set of ready operations; if all operations have been scheduled, proceed to the terminated state; and update the disjunction graph, changing the state from [previous state]. Transition to .

3. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 2, characterized in that, In step 1.1, each operation Features include: 1) Scheduling flag: 1 for scheduled tasks, 0 for unscheduled tasks; 2) Minimum processing time on compatible machines; 3) Average processing time on compatible machines; 4) Remaining processing time; 5) Waiting time, currently unschedulable operations are recorded as 0; 6) Processable Operations The percentage of machines; 7) Related Tasks Unscheduled operands in; 8) Related tasks The remaining processing time, operation The average processing time of unscheduled operations; Each machine Features include: 1) Free space flag: 0 for free space, 1 for not free space; 2) Compatible unscheduled operands; 3) The currently available candidate operands; 4) The shortest processing time for all compatible unscheduled operations; 5) Average processing time for all compatible unscheduled operations; 6) Expected idle time; 7) Waiting time; 8) Utilization rate, the ratio of non-idle time to total production time; Each compatible operation—machine pair The features include the following information: 1) Processing time ; 2) Processing time and The ratio of maximum processing time; 3) Processing time and The ratio of the maximum processing time for compatible unscheduled operations; 4) Processing time The ratio of the maximum processing time to the maximum processing time of a compatible operating-machine pair; 5) Processing time and The ratio of the remaining processing time; 6) and The sum of waiting times.

4. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 2, characterized in that, Step 2 is as follows: Step 2.1: Construct a temporally decoupled operation attention mechanism. By introducing a trainable bias, the neighbors of the operation node are clearly distinguished as predecessors and successors, thereby achieving decoupling and explicit modeling of the temporal relationship between operations. Step 2.2: Construct a dynamic edge-enhanced machine attention mechanism; Step 2.3: Construct a multi-head attention mechanism for inter-head interaction; Step 2.4: Construct a cross-attention mechanism to fully couple the different dimensional features captured by GAT and multi-head attention mechanisms; Step 2.5: Through linear layer adaptive fusion, operational features for enhanced fusion are formed, and pooling is performed on the features after cross-attention fusion to obtain auxiliary global features.

5. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 2, characterized in that, Step 2.1 is as follows: Through calculation operations Attention coefficients with neighboring nodes are used to obtain operational relevance and identify key operations. Considering that for each operation node, its neighboring nodes are divided into predecessor and successor nodes, for input features... operation nodes The attention coefficients between the node and its predecessor and successor nodes are calculated as follows: in, and Represents a linear transformation; Indicates the activation function; For bias, as shown below: in, , For the attention coefficient, the standardized attention coefficient is obtained by normalization using the Softmax function. : Finally, a weighted linear combination of the features after linear transformation of neighboring nodes is performed, and a Sigmoid activation function is applied. Obtain the operational feature vector: 。 6. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 5, characterized in that, Step 2.2 is as follows: In the production process, machines capable of handling the same operation are called competing machines. These competing machines compete for unscheduled operations that they are compatible with. The intensity of this competition varies among these machines. , indicating machine and machines The intensity of competition between them; based on the operational characteristic vectors of the competing machines. To determine the intensity of competition, a lightweight multilayer perceptron (MLP) is introduced to map the original edge features to dynamic weights of 0-1, realizing a machine attention mechanism for dynamic edge enhancement; specifically, for the machine... The original competition vector First, the weights are generated by passing through two layers of feedforward network and using Sigmoid activation: in, The Sigmoid activation ensures the output falls within the [0, 1] interval, acting as a gating mechanism; subsequently, dynamic weights are used to enhance edge features, achieving context-aware modulation. in, It is a calibration The weight matrix is ​​then used, and finally, the modulated edge features are concatenated with the machine node features to obtain the inter-machine attention coefficients. in, It is a calibration and The weight matrix, Represent a linear transformation; , They are machines and machines Node characteristics; Subsequently, by performing Softmax normalization, weighted combination, and activation on the attention coefficients, the final machine feature vector is obtained. : in, Is with A set of machines that are in competition with each other.

7. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 6, characterized in that, Step 2.3 is as follows: Construct a multi-head attention mechanism for inter-head interaction, with a set number of attention heads. First use The attention heads with independent parameters are computed in parallel to obtain... Equal-dimensional features Then, following the aggregation strategy of the GAT architecture, except for the last layer which uses an averaging operator, the remaining layers are spliced ​​together, and the spliced ​​or averaged results are then processed. Send to the head-to-head interaction layer to generate for each location Dynamic weights Next, the multi-head results are weighted and summed using dynamic weights to obtain the weighted fused features. Finally, the activation function is used. Output the final weighted fused features: 。 8. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 7, characterized in that, Steps 2.4 and 2.5 are as follows: Step 2.4: Construct a cross-attention mechanism to fully couple the different dimensional features captured by GAT and multi-head attention mechanisms; [The text abruptly ends here, likely due to an incomplete sentence or missing information.] The outputs after the layer network are denoted as follows: Using local features of the GAT as queries and global features of the Transformer as keys and values, this approach maintains the dominance of local priority information in the current scheduling state while ensuring accurate injection of global information. The features fused through cross-attention are denoted as... and ; Step 2.5: Stitching and Pooling Through linear layer adaptive fusion, operational features for enhanced fusion are formed. and machine characteristics ; Furthermore, pooling is performed on the features after cross-attention fusion to obtain auxiliary global features: in, , These represent the sets of operations and machines, respectively.

9. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 8, characterized in that, Step 3 is as follows: Step 3.1: Building an actor network Its neural network parameters are First, the extracted operational features, machine features, auxiliary global features, and the calculated operational-machine pair features are concatenated and input into the actor network to generate each action in the following manner. Corresponding scalar : Then, the probability distribution for selecting each action is output using the Softmax function: Step 3.2: Building a network of judges Using pooled auxiliary global features As input, a scalar is generated as an estimate of the state value.

10. The flexible job shop scheduling method based on PAH-Attn-DRL according to claim 9, characterized in that, Step 4 specifically includes: Step 4.1: Initialize hyperparameters Set the experience replay pool size and the minimum batch size. b Set the neural network learning rate, set the hidden layer dimension, and set the discount rate. The number of GAT and Transformer layers in both the operational and machine feature extraction networks is set to the same number. Step 4.2: Based on the state dimension in the flexible workshop scheduling environment, set the input dimension of the actor network in Step 3.1, set the output dimension based on the number of machine pairs in the environment, set the activation function to ReLU, and perform SoftMax normalization on the output results; based on the observation dimension, set the input dimension of the judge network in Step 3.2, set the network output dimension to 1, and set the activation function to ReLU. Step 4.3: Receive the state given by the flexible workshop scheduling environment at the start of each round, as input to the actor network and the judge network. The actor network outputs the action probability value of the agent in the environment. If the action is not feasible, its probability is set to 0. The judge network estimates the corresponding state value. The agent refers to the scheduling decision-maker or scheduler. Step 4.4: After each time step in the flexible workshop scheduling scenario, collect the current state information, actions, rewards, and the next state information. Stored in the experience replay pool, awaiting training; Step 4.5: During training, a minimum batch is extracted from the data in the experience replay pool, and the network is updated using the Adam optimizer to minimize the composite loss, which includes policy gradient pruning, value function, and entropy regularization. After training, a flexible job shop scheduling method model based on PAH-Attn-DRL is obtained.

Images