Flexible job shop scheduling method based on preference driven graph reinforcement learning

Abstract

The embodiment of the invention discloses a flexible job shop scheduling method based on preference-driven graph reinforcement learning, and relates to the field of shop dynamic scheduling in an intelligent manufacturing technology. According to the method, by constructing a multi-objective optimization model, multiple objectives of the job shop can be optimized at the same time, including the minimum completion time, the total delay and the total cost. Wherein an imperfect maintenance model is constructed, and the maintenance demand and the maintenance opportunity of each machine are dynamically determined. And capturing a complex relationship between the operation and the machine by using an improved graph neural network. In combination with a preference-driven mechanism, a maintenance plan and workshop scheduling are adjusted in real time through a graph reinforcement learning method, and efficient priority scheduling rules under different preferences are learned, so that an integrated decision of machine allocation, an operation sequence and maintenance arrangement in dynamic scheduling is realized.

Description

Technical Field

[0001] This invention relates to the field of dynamic workshop scheduling in intelligent manufacturing technology, and in particular to a flexible workshop scheduling method based on preference-driven graph reinforcement learning. Background Technology

[0002] The Flexible Job Shop Scheduling Problem (FJSP) is one of the most prevalent production scheduling problems. FJSP is designed for more dynamic and complex job shop environments, better reflecting actual production needs. Especially in actual production processes, operations may be interrupted due to various factors such as changes in work tasks and machines. Incorporating these dynamic events into the FJSP and constructing a dynamic FJSP makes the problem more realistic, but it also increases its complexity. Therefore, developing online scheduling techniques adapted to the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is crucial for handling uncertain events.

[0003] Two common dynamic events in production are the arrival of new jobs and machine failures. To overcome these sudden interruptions, some studies have adopted mainstream methods to solve scheduling problems, such as heuristic and metaheuristic methods. Heuristic methods can react quickly to uncertain events and are highly efficient, but their universality is low due to the diversity of scheduling rules for different problems and production goals, resulting in inconsistent solution quality. Metaheuristic techniques improve solution quality through iterative and parallel search. However, due to their lower complexity, they are difficult to meet real-time requirements in actual production. Rescheduling is required when dynamic events occur, which is difficult to implement in actual production. Therefore, some researchers use reinforcement learning to solve DFJSP, which is specifically understood as finding the optimal action at each decision point and responding to dynamic events in real time. DFJSP scheduling can be modeled as a Markov decision process, providing an ideal framework for deep reinforcement learning. It is well known that maintenance is an important means to ensure the long-term reliable operation of industrial machinery, and a reasonable maintenance plan can reduce the overall operating cost by 20% to 30%. Machine failures require immediate reactive corrective maintenance. In contrast, another maintenance strategy—preventive maintenance—is more popular and widely used as a proactive maintenance strategy. Preventive maintenance (PM) activities include time-based maintenance and condition-based maintenance. During shop floor processing, machine failures have a certain probability of occurrence. To prevent these failures, PM can be implemented proactively, thereby avoiding increased costs and extended production cycles due to failures. Currently, it is assumed that maintenance can prevent unexpected machine failures. Although PM can prevent unexpected interruptions and extend machine life, unexpected machine failures can still occur, leading to machine unavailability and reduced uptime.

[0004] The main reason these existing solutions have various problems is that most research based on Deep Reinforcement Learning (DRL) focuses on single-objective dynamic problems, such as studying processing time / completion time as a single objective. This research approach mainly relies on rule generalization. However, in practical applications, to make the problem more consistent with the actual production environment, it is necessary to consider multiple objective factors simultaneously. Existing technologies rely on insufficient rule generalization and cannot generate solution sets with preference conditions, thus failing to effectively solve problems under multiple objective factors and failing to ensure optimal scheduling decisions in different production environments. Summary of the Invention

[0005] The embodiments of the present invention provide a flexible job shop scheduling method based on preference-driven graph reinforcement learning, which can effectively solve the problem under multiple objective factors and ensure optimal scheduling decisions are achieved in different production environments.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] Establish a multi-objective optimization model for the corresponding flexible operation workshop;

[0008] The process involves acquiring the job set and machine set corresponding to the actual production situation and transforming them into a heterogeneous graph. The machine set includes the number of machines in the flexible workshop, and the job set includes the number of jobs being processed and the number of jobs to be processed. Based on the actual production situation, the job set and machine set are acquired and transformed into a heterogeneous graph. Initial features of machine nodes, operator nodes, and operator-machine (OM) pairs are defined. An attention-based model is then applied to the heterogeneous graph. This model incorporates the embedding process of machines and operator nodes, considering the competitive relationships between machines and the processing relationships between machines and processes to characterize the workshop state.

[0009] The multi-objective optimization model is decomposed, and a multi-objective Markov decision process is constructed.

[0010] Upon the arrival of a new order, the heterogeneous graph structure is updated, and a decision scheme for dynamic workshop scheduling is obtained through the multi-objective Markov decision process. Specifically, for the arrival of a new order, the heterogeneous graph structure is updated, the workshop state is obtained based on a multi-head attention mechanism, and a decision scheme is obtained according to the multi-objective Markov decision process to complete the dynamic workshop scheduling under the condition of random order arrival.

[0011] Specifically, the optimization objectives of the multi-objective optimization model include at least: minimizing the total workpiece delay (TWT) and the maximum workpiece completion time (C). max Minimize both processing and maintenance costs; at time zero, the flexible workshop has a set of n jobs, denoted as J = {J1, J2, ..., J...} n} and a set containing m machines, denoted as M = {M1, M2, ..., M} m Each assignment J i ∈J are all composed of a series of operations with precedence constraints. Composition. Each operation O ij (i.e., the j-th operation of the i-th job) can choose a machine belonging to its set of available machines. Machine M k Execution, the corresponding processing time is p ijk In a dynamic environment of new job arrivals and machine maintenance, new job n1 will arrive dynamically online. For the i-th new job J... i Its arrival time, delivery deadline, and urgency level are respectively A i D i and Pr iIt is important to note that the higher the urgency level, the greater the penalty for delay. Furthermore, maintenance planning is a crucial component of scheduling. Preventive maintenance (PM) needs to be handled promptly to avoid machine failures; while repair (MR) needs to be completed during processing, such as lubricating machines or similar operations. The workshop multi-objective optimization model considering machine maintenance and the arrival of new workpieces is as follows, with the maximum completion time max(C) as the criterion. pk The optimization model is constructed with the following objectives: minimizing the total cost (i.e., processing cost + maintenance cost) and minimizing the cumulative delay (TWT).

[0012]

[0013] C pk ≤S p(k+1) ,p=1,2,...,n;k=1,2,...,h p

[0014] T is ≤B i(s+1) ,i=1,2,...,m; s=1,2,...,n i

[0015]

[0016] C max =max(C pk p = 1, 2, ..., n; k = h p

[0017]

[0018] Where min and max represent the minimum and maximum functions, respectively, O pk S represents the k-th process of the p-th workpiece, where p and k are both positive integers; pk For process O pk Start time; C pk For process O pk The completion time; s is the processing position identifier of the process on the same equipment; B is T represents the start time of the operation at the s-th processing position on the i-th machine, where i is a positive integer; is x represents the process completion time at the s-th processing position on the i-th machine; bais As a marker, when process O ba When processing at the s-th processing position of the i-th machine, x bais =1 otherwise 0; n i Z represents the total number of processes performed on the i-th machine; isZ is the marker value for preventative maintenance. is =1 indicates that preventative maintenance (PM) is required; D is p represents the availability parameter of the i-th device before starting the operation at the s-th processing position; pkis For process O pk The total cost for processing time on the i-th machine includes preventative maintenance (PM), minor repairs (MR), and operating costs. b Let x represent the number of operations in the p-th job. pki(s-1) This indicates whether the k-th operation of job p is assigned to the (s-1)-th position on a certain machine, tp i h represents the time spent on each preventative maintenance of machine i. i S represents the total number of processes contained in workpiece i. ba Indicates process O ba The start processing time, p iba Indicates process O ba The processing time on the i-th machine is N. is t represents the number of minor repairs performed on the i-th machine at the s-th position. M Let represent the cumulative maintenance or processing time for the Mth machine, and 'c' represent the cost variable (used to represent maintenance or processing costs). Assume that the cost of a single PM and minor repair for each machine is fixed (cpi and cr, respectively). i ), np i Let E[nr] be the total number of PMs for the i-th machine. i Let be the total number of minor repairs for the i-th machine. At time zero, the flexible work workshop contains a set of n jobs, and there are m machines in the flexible work workshop; C pk For process O pk The completion time, where p represents the workpiece number and k represents the process number; h j C represents the total number of processes contained in workpiece j; p Indicates the completion time of process p; the j-th operation O of the i-th job. ij Machine M k Execution, O ij The corresponding processing time is p ijk cost M This indicates the cost of preventative maintenance and minor repairs. This indicates the last process of process i (the hth process). i (Tao) Completion Time; D i Indicates the delivery date of process i; Pr i This indicates the priority of process i.

[0019] The process of transforming heterogeneous graphs includes: setting initial features for machine nodes, operation nodes, and operation-machine (OM) pairs in the heterogeneous graph. The heterogeneous graph model is H = (O, M, C, ε), where O is the process node type, M is the machine node, C is the set of directed edges representing the order constraints between processes, and ε represents the edge type. Specifically, the job set and machine set are obtained based on actual production conditions and transformed into a heterogeneous graph. After defining the initial features for machine nodes, operation nodes, and operation-machine (OM) pairs, step S2 proposes a heterogeneous graph model H = (O, M, C, ε), which expands the process node type O of the traditional disjunctive graph by adding machine nodes M, expanding the edge type ε, and adding edge information. Based on the heterogeneous graph, considering the arrival factor of dynamic workpieces, a time factor is introduced. The time information of the sequence not only reflects the sequential relationship of elements in the sequence but also the interval between two events. For example, the arrival time of a new workpiece and the start time t of the processing operation on the machine. The d-dimensional time is defined as the d-dimensional time encoding function corresponding to the heterogeneous graph: d represents the number of dimensions, ω1 represents the frequency component of the first dimension; ω d This represents the frequency component of the d-th dimension, used to control the periodic variation characteristics in different dimensions;

[0020] The heterogeneous diagram represents the workshop status of the flexible work workshop.

[0021] The heterogeneous graph characterizes the workshop state of a flexible workshop, including: embedding process nodes into the heterogeneous graph based on meta-paths, and fusing relationships between process nodes with intermediate nodes. Specifically, a model based on an attention mechanism and a meta-path-based transformer is applied to the heterogeneous graph. This model combines the embedding process of machine and operator nodes, considering competition between machines and processing relationships between machines and processes to characterize the workshop state. Process nodes are embedded based on meta-paths; for nodes without intermediate nodes, a multi-head attention mechanism is used for embedding. ij There are various types of relationships, including the preceding process O ij-1 The next process O ij+1 (If they exist), then the features of the two nodes and The attention coefficient between them is: in, and These are linear transformations. Represented as an attention weight vector, used for the linear combination and concatenation of features; R represents the real number field; d' o d represents the feature dimension of the process node after linear transformation in the attention mechanism. O This represents the dimension representing the characteristics of the original process node; Indicates process node O ij The original feature vector; This indicates the process node O. ip The original feature vector. and These are linear transformations, the transformed matrices are concatenated using the concatenation operator ||, and the results are output using the LeakyReLU activation function. The softmax function is then applied to all attention coefficients e. i,j,p Normalization is then performed. Finally, the code for a process node without a metapath is obtained through linear fusion with its neighbor node.

[0022]

[0023] in, Indicates process node O ij The final embedding vector; σ represents the activation function, using LeakyReL; α i,j,p Indicates process node O ij With process node O ip Attention relevance score; encoding machine nodes in the heterogeneous graph using a graph attention network; obtaining global features of the heterogeneous graph. and These are linear transformations, the transformed matrices are concatenated using the concatenation operator ||, and the results are output using the LeakyReLU activation function. The softmax function is then applied to all attention coefficients e. i,j,p Normalization is then performed. Finally, the code for a process node without a metapath is obtained by linearly fusing it with its neighbor node.

[0024] For merging relationships between process nodes with intermediate nodes, it is necessary to aggregate the meta-paths, incorporating intermediate node and edge features, to achieve meta-path O. ij -M k -O pl For example, define R ijk For node O ij and M k The edge relationships between them, and the characteristics of the starting node of the path are: The cumulative features of the metapath are as follows: The final meta-path feature is The query vectors for the attention mechanism are obtained sequentially. key vector Sum value vector Calculate the attention coefficient to aggregate path features. This represents the query transformation matrix corresponding to the l-th layer; Represents the target process node O in the l-th layer. ij Embedded representation; This represents the bias term used to generate the query vector in the l-th layer; This represents the key transformation matrix corresponding to the l-th layer; This represents the bias term used to generate the key vector in the l-th layer; h represents the value transformation matrix corresponding to the l-th layer; P(i,j) The feature vector representing the path P(i,j) from node i to j; Let represent the bias term of the generated value vector in layer l:

[0025]

[0026] Where c is the number of heads in the multi-head attention mechanism, exp is the exponential function, and Concat is the concatenation function. The attention score represents the relationship between the target process node i and the path P(i,j) from node i to j in the l-th layer. The normalized attention weight represents the degree of influence of path P(i,j) on target node i. This represents the path aggregation representation of the c-th attention head to node i in layer l; p represents the path index number (which can represent a path from i to j); N ( ij Let ) represent the set of available paths for node pair P(i,j);

[0027] This represents the attention weight of the c-th attention head in layer l for path P(i,j); Let h represent the bond transformation matrix corresponding to the l-th layer; P(i,j) The feature vector representing the path P(i,j) from node i to j; This represents the bias term used to generate the key vector in the l-th layer; H represents the final embedding representation of node i in layer l+1; H represents the total number of heads in multi-head attention (i.e., the number of splices); This represents the aggregated output of the H-th attention head in the l-th layer on the path P(i,j) for node i;

[0028] To prevent the model from becoming overly smooth, residual connections are added. To avoid gradient vanishing in deep networks while preserving input feature information, the input features from the previous layer are directly added to the output. The final output of the entire meta-path-based Transformer layer still needs to pass through a regularization layer, residual connections, and a feedforward neural network.

[0029]

[0030] This represents the node representation after residual connection and normalization (LayerNorm); This represents the original input features of node i in layer l; This represents the output of the path attention (or aggregation module) of node i in layer l+1; This represents the intermediate representation of node i in layer (l+1) after processing by the feedforward neural network (i.e., the output after linear mapping and activation function processing). This represents the second set of weight matrices in the feedforward neural network of layer l; This represents the first set of weight matrices in the feedforward neural network of layer l; Indicates…; l represents the Transformer layer number currently in the network; l is used to indicate the next layer number after the current Transformer layer in the network;

[0031] Aggregated information from different meta-paths is input into the weighted feature fusion module to complete the fusion of information from different meta-paths. Here, c1 and c2 are both adjustable hyperparameters. This represents the feature representation of node i at layer l+1 after aggregation from neighborhood paths (such as directly connected nodes); The feature representation of node i at layer l+1 after aggregation of intermediate node paths (such as heterogeneous paths); l represents the next layer after layer l; in the subsequent implementations of this paper, the values ​​are set to 0.6 and 0.4 respectively, assuming that the importance of neighboring nodes is greater than that of having intermediate nodes.

[0032]

[0033] The machine nodes employ a graph attention-based network, including domain process nodes and machine nodes with competing relationships. Define machine m. p The set of operations that can be processed C p Each machine connects to only one OM pair, which is then connected to the operation set C. p Connected. Attention coefficients are calculated using two linear transformations.

[0034] and To process machine nodes and operation nodes, the machine nodes in the heterogeneous graph are encoded using a graph attention network. Specifically, the machine node encoding process uses a graph attention network, including neighborhood operation nodes and machine nodes with competing relationships. Define machine m. p The set of operations that can be processed C p Each machine connects to only one OM pair, which is then connected to the operation set C. p Connected. Attention coefficients are calculated using two linear transformations. and To process machine nodes and operation nodes As shown below: Among them, u k,m,n This represents machine node m. k With operation node O mn Attention coefficient between, Represents a machine node. Indicates the operation node. Represents the attention weight vector. d m This represents the dimension of the node's embedded features. To further consider the competition between different machine nodes when processing the same operation node, NO(m p ,m q Defined as machine m p and m q The set of competing operations, defined as the set of machines competing with machine mk. As machine m p and m q The performance metric for competition analysis between processes means that the machine prioritizes the operation that competes most fiercely with the unscheduled process. If NO(m k ,m q If ∩J is empty, then the relevant matrix is ​​represented as zero, and J represents the index matrix of the competition relationship between machines.

[0035] in, To further consider the competition between different machine nodes when processing the same operation node, let machine m be defined as... p and m q The set of competing operations, defined as the set of operations with machine m. k Competing machine set NO(m p ,m q As machine m p and m q The performance metric for competition analysis between operations means that the machine prioritizes the operation that competes most fiercely with the unscheduled operation. If NO(m k ,m q If ∩J is empty, then the relevant matrix is ​​represented as zero. For each machine M k ∈M has the original characteristics All attention coefficients u kq Calculate as follows:

[0036] Among them, u kq Represents machine node m k With operation node m q Attention coefficient between, X1 and X2 represent three different linear transformation matrices. Represents machine node m q Features, oc kq This indicates a competition relationship between two machine nodes;

[0037] and These are three different linear transformation matrices. (Specially, oc) kk It can be used to measure the processing power of machine Mk without considering its own competition; the attention coefficient u kk The calculation is as follows: Among them, u kk This represents the attention coefficient of the node itself.

[0038] right and u kk Normalization is performed to obtain the corresponding normalized attention coefficients.

[0039] β k,m,n ,β kq and β kk ;

[0040] Then, for u k,m,n ,u kq and u kk Normalization is performed to obtain the normalized attention coefficient β. k,m,n ,β kq and β kk By using the softmax function. Finally, by combining the data NO(M) from nearby operations. k M q The machine itself, and the embedding of computer machine nodes:

[0041] This represents the updated machine node embedding vector, where σ represents the non-linear activation function (such as ReLU or LeakyReLU). Indicates the current machine node m k The original embedding vector, O ij Let m represent the j-th operation node of job i. k Let m represent the k-th machine node. q Indicates with m k Other machine nodes that are in competition, Z2 represents the linear transformation matrix acting on the operating node, and q represents the index of the adjacent machine node. This indicates that there is a competing relationship between machine nodes m. q The original embedding vector.

[0042] Global feature acquisition. A graph-level feature set is constructed by fusing the node features from each layer. The original features of operation node Oij and machine node Mk are represented as follows: and The aforementioned embedding process can be viewed as an attention layer, passed through N attention layers to acquire learned features. and Average pooling is applied to the operational features and machine features separately. The concatenated result generates a global feature set for a MODFJSP instance. The process of obtaining the global features of the heterogeneous graph includes:

[0043] in, Represents the global features of a MODFJSP instance, where N represents the number of layers in the graph neural network, and O represents the global features of the instance. u Let M represent the set of all operation nodes in the graph. u This represents the set of all machine nodes in the graph.

[0044] The decomposition strategy for the multi-objective optimization model uses the workshop state and preference weights as the state space and the machine selection process as the action space. Specifically, the decomposition strategy simplifies the multi-objective dynamic optimization problem into multiple single-objective dynamic optimization problems by using the weight vector w∈[0,1] and the combination function W(·), with the combination function being a linear weighting function. Using the aforementioned machine node features, graph-level features, and weight vectors as the state space, and based on the aforementioned heterogeneous graph representation method, a decision network is designed based on multi-objective SAC, including process strategies. and machine strategy Specifically, for each operation a ot ∈A o and machine action a mt ∈A m It produces a scalar μ(a) o |s) and μ(a m |s), and use the softmax function to generate the desired distribution as the output. At decision time t, for each operation or each machine, the operation... or machine Extracting features, global features and compatible OM features These are connected to capture new results of features. These results are then fed into a multi-layer perceptron (MLP) to determine the state S. t The priority of each selected action. The decomposition strategy includes: process strategy. and machine strategy

[0045] The extracted features represent the operation actions. Extracted features representing machine actions This represents the global features of the graph, and represents the representation of the operation actions extracted by the Nth layer of the graph neural network. h represents the representation of the machine action extracted by the Nth layer of the graph neural network. O-M This represents the embedding of the interaction relationship between operations – machine pairs. By considering all a... ot ∈A o and a mt ∈A m Using the softmax function, the selection action μ(a) is obtained. o |s) and machine action μ(a m The probability of |s):

[0046]

[0047] μ(a o |s) represents the operation action a ot The corresponding scalar, μ(a) m |s) represents machine action a mt The corresponding scalar, a ot ∈A o a mt ∈A m A o A represents the motion space of a process. m a represents the machine's motion space. o a represents an action in the set of optional process actions. m θ represents an action from the set of available machines for the current process. o Represents the process strategy network, θ m This represents the machine policy network, where s represents the current state and b represents the current state. o b represents the alternative actions in the action space for the corresponding process selection. m This represents the alternative actions in the corresponding machine's action space, and μ represents the scoring function;

[0048] The step of decomposing the multi-objective optimization model includes: decomposing the multi-objective into three objectives, wherein, for the maximum completion time, the immediate reward R1 = C is decomposed. max (t-1)-C max (t); For the total cumulative delay TWT, decompose it into the instant reward R2 = TWT ave (t-1)-TWT ave (t); For the cumulative cost, decompose it into the immediate reward R3 = Cost.ratio (t-1)-Cost ratio (t); where C max (t) represents the maximum completion time at the current moment, TWT ave (t) represents the average total delay time at the current moment, and Cost ratio (t) represents the cost percentage at the current moment;

[0049] The weighted average of the three objectives is R. t =ω1r1 + ω2r2 + ω3r3, where r1, r2, and r3 represent the immediate rewards of the corresponding sub-goals, and ω1, ω2, and ω3 represent the corresponding weighting coefficients. 2+ ω3=1.

[0050] In Markov decision-making, the reward function, Rt, is determined by evaluating whether the indicators corresponding to the three objectives have improved compared to time t-1. For the maximum completion time, the immediate reward R1 is C. max (t-1)-C max (t), for the total accumulated delay TWT, the instant reward R2 is TWT. ave (t-1)-TWT ave (t), for the cumulative cost timely reward R3, is the ratio of shop floor running time to total cost (Cost). ratio (t-1)-Cost ratio (t). To ensure that all objectives can be optimized, the three objectives will be summed in a weighted manner, as shown by the formula R. t =ω1r1+ω2r2+ω3r3 and ω1+ω 2+ ω3 = 1. Since the reward functions for the two target dimensions have different units, a normalization method is used to process the reward vectors to make them additivity. The specific formula is as follows: The reward vectors are normalized as follows: R max,l,t R min,l,t R represents the maximum and minimum values ​​of the l-th objective function in time interval t, respectively. 1,t R 2,t and R 3,t These are the reward values ​​for the first, second, and third objectives at time t, respectively.

[0051] Furthermore, the process of updating the heterogeneous graph structure also includes: updating the heterogeneous graph structure based on the maintenance status of the machines and equipment in the flexible work workshop. For example: for equipment failures, an imperfect maintenance model of the equipment is constructed based on the Welb distribution to determine whether the equipment needs maintenance and the type of maintenance, thus updating the heterogeneous graph structure, and constructing a maintenance agent to select the type of maintenance and whether maintenance is required. For example:

[0052] For changes in the graph structure, newly arrived workpieces are added to the graph structure according to their process sequence, and the connections between nodes are based on processing relationships. For maintenance model construction, the Weibull distribution is used to analyze the machine's service life and determine whether maintenance is required. A two-parameter Weibull distribution is used to model the machine's failure modes, and the failure probability density can be expressed as:

[0053]

[0054] Where β>0, θ>0, β and θ are the shape parameter and size parameter respectively, T is the service life of the machine, and the failure rate is as follows:

[0055]

[0056] The expected number of failures E[N(ω)] within each time unit ω, and corrective maintenance (CM) is performed when random failures occur:

[0057]

[0058] N(ω) represents the number of random failures within time ω, where ω represents the upper limit of time, i.e., one analysis period / time unit, and λ(T) represents the failure rate function at time T.

[0059] Preventive maintenance (PM) can only improve the condition and performance of a machine to a certain extent, but it cannot restore it to a brand-new state. Therefore, improvement factors are introduced. To describe the impact of PM on machine status:

[0060]

[0061] Where I is the machine number, representing the i-th machine, cp I and ca I This refers to the replacement cost and PM cost of the I-th machine. Based on expert experience, the cost adjustment factor u, time adjustment factor v, cumulative maintenance frequency b, and learning effect adjustment factor w are used. m represents the total number of machines, and np... ILet represent the maximum number of PMs that the i-th machine can perform. Therefore, the effective service life of the i-th machine before and after processing the s-th location can be expressed as:

[0062]

[0063] s = 1, 2, 3, ..., n I ;I=1,2,3,...,m; b=1,2,...,np I

[0064]

[0065] Where, n i It represents the number of operations processed on the i-th machine, np. I Let m be the number of PMs on the i-th machine, and m be the number of machines. The reliability D of the i-th machine before processing the s-th position operation. is Calculated according to equation (7), and compared with the reliability threshold of the i-th machine. Compare them. If the reliability is greater than or equal to... The machine should then perform PM and generate PM time. Otherwise, the PM time will be 0. a represents the remaining effective lifespan of the machine calculated up to the current operating position. Is p represents the actual lifetime value at the s-th position of the current machine. Ija The processing time is represented by s, where s represents the s-th operation being processed by machine I.

[0066] This invention, by combining an imperfect maintenance model, heterogeneous graph representation, preference-driven mechanisms, and decomposition strategies, not only optimizes multiple objectives such as shop floor completion time, total delay, and total cost, but also responds in real time to dynamic events such as new job arrivals and machine failures, improving the efficiency of shop floor resource utilization. Utilizing graph reinforcement learning algorithms, scheduling strategies can be intelligently adjusted to ensure optimal scheduling decisions under different production environments, demonstrating high practicality and application value. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of the method of the present invention.

[0069] Figure 2 This is a model diagram for constructing heterogeneous graphs under undisturbed and dynamic events conditions according to the present invention.

[0070] Figure 3 This is a network diagram of the graph embedding process of the present invention.

[0071] Figure 4 This is the construction module for the multi-objective dynamic scheduling system based on graph reinforcement learning of the present invention.

[0072] Figure 5 This is a schematic diagram of the MDP decision-making process in an embodiment of the present invention.

[0073] Figure 6 The figure shows the comparison results of the implementation case of this invention with other priority scheduling rules and other reinforcement learning methods.

[0074] Figure 7 This is a comparison diagram of the solution example of this invention with other maintenance rules. Detailed Implementation

[0075] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Embodiments of the present invention will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of the present invention means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0076] like Figure 1As shown in the embodiment of the present invention, the dynamic multi-objective flexible job shop scheduling method based on preference-driven graph reinforcement learning proposed in this invention is applied to the automatic control system of the flexible job shop, and includes the following steps:

[0077] 1) Construct a multi-objective optimization model to minimize the total workpiece delay time, the maximum workpiece completion time, and the processing and maintenance costs.

[0078] 2) Obtain the job set and machine set based on the actual production situation, and transform them into a heterogeneous graph. Define the initial characteristics of machine nodes, operation nodes and operation-machine (OM) pairs.

[0079] 3) An attention-based model is applied to heterogeneous graphs. This model combines the embedding process of machines and operation nodes, and considers the competitive relationship between machines and the processing relationship between machines and processes to characterize the workshop state.

[0080] 4) The multi-objective problem is decomposed into an optimization model with preference weights based on the decomposition strategy. The workshop state and preference weights are used as the state space, and the selection of process machine pairs is used as the action space to construct a multi-objective Markov decision process.

[0081] 5) For the arrival of new orders, update the heterogeneous graph structure, obtain the workshop status based on the multi-head attention mechanism, obtain the decision scheme according to the multi-objective Markov decision process, and complete the dynamic scheduling of the workshop under the random arrival of orders.

[0082] 6) For equipment failures, an imperfect maintenance model for the equipment is constructed based on the Welb distribution to determine whether the equipment needs maintenance and the type of maintenance, update the heterogeneous graph structure, and construct a maintenance agent to select the type of maintenance and whether maintenance is required.

[0083] Furthermore, at time zero, the flexible workshop has a set containing n jobs, denoted as J = {J1, J2, ..., J...} n} and a set containing m machines, denoted as M = {M1, M2, ..., M} m Each assignment J i ∈J are all composed of a series of operations with precedence constraints. Composition. Each operation O ij (i.e., the j-th operation of the i-th job) can choose a machine belonging to its set of available machines. Machine M k Execution, the corresponding processing time is p i,j,k In a dynamic environment of new job arrivals and machine maintenance, new job n1 will arrive dynamically online. For the i-th new job J... i Its arrival time, delivery deadline, and urgency level are respectively Ai D i and Pr i It is important to note that the higher the urgency level, the greater the penalty for delay. Furthermore, maintenance planning is a crucial component of scheduling. Preventive maintenance (PM) needs to be handled promptly to avoid machine failures; while repair (MR) needs to be completed during processing, such as lubricating machines or similar operations. The workshop multi-objective optimization model considering machine maintenance and the arrival of new workpieces is as follows, with the maximum completion time max(C) as the criterion. pk The optimization model is constructed with the following objectives: minimizing total cost (Cost) and cumulative delay (TWT).

[0084]

[0085] C pk ≤S p(k+1) ,p=1,2,…,n;k=1,2,…,h p

[0086] T is ≤B i(s+1) ,i=1,2,...,m; s=1,2,...,n i

[0087]

[0088] C max =max(C pk p = 1, 2, ..., n; k = h p

[0089]

[0090] Where min and max represent the minimum and maximum values ​​of the function, respectively; O pk S represents the k-th process of the p-th workpiece; pk For process O p The start time of k; C pk Process O pk Completion time; s Processing position identifier of the process on the same equipment; B is T represents the start time of the operation at the s-th processing position on the i-th machine; is x represents the process completion time at the s-th processing position on the i-th machine; bais When process O ba The value is 1 when processing is performed at the s-th processing position of the i-th device; otherwise, it is 0. i Z represents the total number of processes performed on the i-th machine; is =1 indicates that preventative maintenance (PM) is required; D isThe availability of the i-th machine before the start of the s-th processing position; p pkis For process O pk The processing time on the i-th machine; the total cost includes preventative maintenance (PM), minor repairs (MR), and operating costs. Assume that the cost of a single PM and minor repair per machine is fixed (cp respectively). i and cr i Let np be the total number of PMs and the total number of minor repairs for the i-th machine. i and E[nr i ].

[0091] Furthermore, the disjunctive graph is improved into a heterogeneous graph to represent the workshop state. Since the disjunctive graph model of FJSP is relatively complex, a heterogeneous graph model H = (O, M, C, ε) is proposed. This model expands the traditional disjunctive graph by adding machine nodes (M) and edge types (ε), and also adds edge information. The initial heterogeneous graph is as follows: Figure 2 As shown in (a). Based on the heterogeneous graph, considering the arrival factors of dynamic workpieces, a time factor is introduced. The time information of the sequence can reflect not only the order of elements in the sequence, but also the interval between two events. For example, the arrival time of a new workpiece and the start time of the processing step on the machine. The d-dimensional time encoding function is defined as:

[0092]

[0093] Furthermore, the specific methods for representing the workshop state include machine node embedding, process node embedding, and global graph-level feature acquisition. The specific embedding process is as follows: Figure 3 As shown. The specific process is as follows:

[0094] 1) The process nodes are embedded based on meta-paths. For nodes without intermediate nodes, a multi-head attention mechanism is used for embedding. Operation O ij There are various types of relationships, including the preceding process O ij-1 The next process O ij+1 (If it exists), then the attention coefficient between the two nodes is:

[0095]

[0096] in, and These are linear transformations, the transformed matrices are concatenated using the concatenation operator ||, and the results are output using the LeakyReLU activation function. The softmax function is then applied to all attention coefficients e. i,j,p Normalization is then performed. Finally, the code for a process node without a metapath is obtained through linear fusion with its neighbor node.

[0097]

[0098] For merging relationships between process nodes with intermediate nodes, it is necessary to aggregate the meta-paths, incorporating intermediate node and edge features, to achieve meta-path O. ij -M k -O pl For example, define R ijk For node O ij and M k The edge relationships between them, and the characteristics of the starting node of the path are: The cumulative features of the metapath are as follows: The final meta-path feature is The query vectors for the attention mechanism are obtained sequentially. key vector and value vector Calculate the attention coefficient to aggregate path features:

[0099]

[0100] Where c is the number of heads in the multi-head attention mechanism, exp is the exponential function, and Concat is the concatenation function.

[0101] To prevent the model from becoming overly smooth, residual connections are added. To avoid gradient vanishing in deep networks while preserving input feature information, the input features from the previous layer are directly added to the output. The final output of the entire meta-path-based Transformer layer still needs to pass through a regularization layer, residual connections, and a feedforward neural network.

[0102]

[0103] The aggregated information from different meta-paths is input into the weighted feature fusion module to complete the fusion of information from different meta-paths, as shown in Equation (3-19). Here, c1 and c2 are both adjustable hyperparameters; in the subsequent implementation of this paper, they are set to 0.6 and 0.4 respectively, assuming that neighboring nodes are more important to the current node than those with intermediate nodes.

[0104]

[0105] 2) The machine node encoding process uses a graph attention-based network, including neighborhood process nodes and competing machine nodes. Define machine m... p The set of operations that can be processed C p Each machine connects to only one OM pair, which is then connected to the operation set C. p Connected. Attention coefficients are calculated using two linear transformations. and To process machine nodes and operation nodes As shown below:

[0106]

[0107] in, To further consider the competition between different machine nodes when processing the same operation node, NO(m p ,m q Defined as machine m p and m q The set of competing operations, defined as the set of machines competing with machine mk. As machine m p and m q The performance metric for competition analysis between processes means that the machine prioritizes the operation that competes most fiercely with the unscheduled process. If NO(m k ,m q If ∩J is empty, then the relevant matrix is ​​represented as zero. For each machine M k ∈M has the original characteristics All attention coefficients u kq Calculate as follows:

[0108]

[0109] in and These are three different linear transformation matrices. kk It can be used to measure the processing power of machine Mk without considering its own competition; the attention coefficient u kk The calculation is as follows:

[0110]

[0111] Then, for u k,m,n ,u kq and u kk Normalization is performed to obtain the normalized attention coefficient β. k,m,n ,β kq and β kk By using the softmax function. Finally, by combining data NO(M) from nearby processes. k M q The machine itself, and the embedding of computer machine nodes:

[0112]

[0113] 3) Global Feature Acquisition. A graph-level feature set is constructed by fusing the node features from each layer. The original features of the operation node Oij and the machine node Mk are represented as follows: and The aforementioned embedding process can be viewed as an attention layer, passed through N attention layers to acquire learned features. and Average pooling is applied to the operational features and machine features separately. The concatenated result generates a global feature set for a MODFJSP instance.

[0114]

[0115] Furthermore, regarding the decomposition strategy in S4, the multi-objective dynamic optimization problem can be simplified into an infinite number of single-objective dynamic optimization problems by using the weight vector w∈[0,1] and the combination function W(·). The combination function used is a linear weighting function. Using the aforementioned machine node features, graph-level features, and weight vectors as the state space, and based on the above heterogeneous graph representation method, a decision network was designed based on multi-objective SAC, including the job strategy. and machine strategy like Figure 4 As shown, specifically, for each operation a ot ∈A o and machine action a mt ∈A m It produces a scalar μ(a) o |s) and μ(a m |s), and use the softmax function to generate the desired distribution as the output. At decision time t, for each operation or each machine, the operation... or machine Extracting features, global features and compatible OM features These are concatenated to capture new results from the features. These results are then fed into the MLP to determine the state S. t The priority of each selected action is as follows:

[0116]

[0117] Then, by applying all a ot ∈A o and a mt ∈A m Using the softmax function, the selection action μ(a) is obtained. o |s) and machine action μ(a m The probability of |s):

[0118]

[0119] In Markov decision-making, the reward function, Rt, is determined by evaluating whether the indicators corresponding to the three objectives have improved compared to time t-1. For the maximum completion time, the immediate reward R1 is C. max (t-1)-C max (t), for the total accumulated delay TWT, the instant reward R2 is TWT. ave (t-1)-TWT ave (t), where the cumulative cost timely reward is the ratio of shop floor running time to total cost (Cost). ratio (t-1)-Cost ratio (t). To ensure that all objectives can be optimized, the three objectives will be summed in a weighted manner, as shown by the formula R. t =ω1r1+ω2r2+ω3r3 and ω1+ω 2+ ω3 = 1. Since the reward functions for the two target dimensions have different units, a normalization method is used to process the reward vectors to make them additivity. The specific formula is as follows:

[0120]

[0121] In the formula: R max,i,t R min,i,t Let ω be the maximum and minimum values ​​of the i-th objective function value during time interval t, respectively, and ω be the values ​​of ω. n =[1,0,0],ω n =[0,1,0] and ω n The objective function values ​​for the cases [0, 0, 1]. R 1,t R 2,t and R 3,t Let be the reward values ​​for the first and second objectives at time t, respectively. These three cases can be solved using the existing single-objective SAC algorithm.

[0122] Furthermore, regarding the changes in the graph structure in S5, newly arriving workpieces are added to the graph structure according to the process sequence, and the connections between nodes are based on processing relationships, such as... Figure 2 As shown in (b).

[0123] Furthermore, for the maintenance model construction in S6, Welb distribution analysis is used to determine the machine's service life, whether maintenance is required, to construct a machine maintenance agent, and to update the heterogeneous graph structure, such as... Figure 2 As shown in (c), the two-parameter Weibull distribution is used to model the machine's failure modes, and the failure probability density can be expressed as:

[0124]

[0125] Where β>0, θ>0, β and θ are the shape parameter and size parameter respectively, t is the service life of the machine, and the failure rate is as follows:

[0126]

[0127] The expected number of failures per ω time unit; if a random failure occurs, a minor repair (CM) will be performed.

[0128]

[0129] Productivity Management (PM) can only improve the condition and performance of a machine to a certain extent, but it cannot restore it to a completely new state. Therefore, an improvement factor is introduced. To describe the impact of PM on machine status:

[0130]

[0131] Where i is the i-th machine, cp i and ca i Let represent the replacement cost and PM cost of the i-th machine. Based on expert experience, the cost adjustment factor u, time adjustment factor v, cumulative maintenance frequency b, and learning effect adjustment factor w are used. Therefore, the effective service life of the i-th machine before and after handling the s-th position can be expressed as:

[0132]

[0133] Where, n i It represents the number of operations processed on the i-th machine, np. i Let m be the number of PMs on the i-th machine, and m be the number of machines. The reliability D of the i-th machine before processing the operation at the s-th location. is Calculated according to equation (7), and compared with the reliability threshold of the i-th machine. Compare them. If the reliability is greater than or equal to... Then the machine should perform PM and generate PM time. Otherwise, the PM time is 0. Based on equation (3), the expected number of random failures that may occur when the i-th machine processes the s-th position operation is given by equation (10):

[0134]

[0135] Ultimately, the system is implemented in the workshop's automatic control system, such as... Figure 4 As shown, it includes a workshop heterogeneous graph construction and feature acquisition module, a node encoding module, an agent decision network construction module, and a decision module.

[0136] The workshop heterogeneous graph construction and feature acquisition module is used to collect data information during the workshop production process, obtain the set of workpieces and machines, construct a heterogeneous graph based on the processing relationship between workpieces and machines, and obtain the initial machine node state, process node state, and machine node pair state characterization of the workshop state from it.

[0137] In the node encoding module, process nodes and machine nodes are embedded using a Transformer layer based on a Graph Attention Network (GAT) and meta-paths. First, process nodes are embedded using a multi-head attention mechanism, calculating attention coefficients based on the relationships between preceding and following processes, and then normalizing them using activation and softmax functions. Finally, encoded information is obtained through linear fusion with domain nodes. For process nodes with intermediate nodes, meta-paths are used for feature aggregation. By calculating the relationships between nodes and edges in the meta-path, an attention mechanism is used to aggregate path features. To avoid gradient vanishing and preserve input feature information, residual connections and regularization layers are added. Machine nodes calculate attention coefficients based on their competition with process nodes, using linear transformations to handle the relationship between machine nodes and operation nodes, and obtaining machine node embeddings through normalization. Finally, global features are generated by fusing the node features from each layer, representing a specific MODFJSP instance.

[0138] The intelligent agent decision network construction module utilizes machine node features, process node features, graph-level features, and weight vectors as the state space. It designs a decision intelligent agent network, including process intelligent agents and machine intelligent agents, through multi-objective SAC.

[0139] In the decision-making module, a machine maintenance agent is constructed based on a two-parameter Welbeck model to determine whether maintenance is needed and the type of maintenance. Regarding the reward function design, optimization is performed by combining three objectives: maximum completion time (Cmax), total cumulative delay (TWT), and cumulative cost (Cost), using a preference space weighted approach to calculate the immediate reward. At each decision time t, the desired action distribution is generated based on the extracted features of the operation or machine action, global features, compatible OM pairs, and preference space features, selecting the machine and the operation. These features are input into a multilayer perceptron (MLP) to determine the priority of each action selected in state St, and the softmax function is used to normalize the priorities of all actions, calculating the probability of selecting the operation action and the machine action.

[0140] Implementation Case 2:

[0141] This invention provides a case study of a dynamic, flexible workshop where workpieces are randomly generated from three types of workpieces (shafts, plates, and flanges). The basic components of the platform include an automated storage and retrieval system (AS / RS, i.e., one warehouse), two lathes (M1, M2), two milling machines (M3, M4), and two drilling machines (M5, M6). The AS / RS is equipped with a warehouse for storing workpiece blanks, and the machine tools are equipped with a buffer for temporarily storing workpieces awaiting processing. The Wilbur distribution parameters for each machine are estimated as shown in Table 1; these values ​​can be obtained by weighted averaging based on the specific details (maintenance type, cost, frequency) of the equipment's most recent PM or MR. The workshop initially has ten workpieces, with ten more arriving subsequently.

[0142] Table 1

[0143]

[0144]

[0145] The method of this invention is compared with eight different composite rules and three different reinforcement learning algorithms, such as Figure 6As shown, several representative scheduling rules were selected, including four job ordering rules (PDR) and two machine allocation rules, and combined into eight composite rules for comparison with the proposed framework. The four job ordering rules are: First-Come, First-Served (FIFO), Earliest Due Time (EDD), Most Remaining Work First (MWKR), and Most Remaining Operations First (MOR). The two machine allocation strategies are: Shortest Processing Time First (SPT) and Earliest Finish Time First (EET). By combining job selection and machine selection rules, eight composite priority scheduling rules (PDR) were derived: FIFO_EET, FIFO_SPT, MOR_EET, MOR_SPT, EDD_EET, EDD_SPT, MWKR_EET, and MWKR_SPT. MOSAC showed the best performance in Cmax reduction rate, remaining at a very low level, indicating its significant optimization effect on maximum completion time. PPO and DDQN followed closely, also demonstrating strong optimization capabilities. The PDR series algorithms (such as PDR1, PDR3, and PDR7) exhibit high reduction rates, demonstrating significant shortcomings and failing to effectively optimize maximum completion time. MOSAC is the optimal scheduling strategy, exhibiting the lowest reduction rate across all objectives (Cmax, TWT, and cost), showcasing its superior performance in complex and dynamic shop floor environments. PPO and DDQN closely follow MOSAC in performance, particularly demonstrating good performance in cost and delay optimization. In contrast, the PDR series algorithms generally have higher reduction rates, particularly showing relatively weak performance on Cmax and TWT objectives, making them unsuitable for multi-objective dynamic optimization problems.

[0146] To further verify the superiority of the machine maintenance agent in this invention, it is compared with the case without a model. To verify the effectiveness of the maintenance model, Figure 7 (a) shows the scheduling results using PM throughout the process. With PM, Cmax, TWT, and cost were 563, 6240, and 1,063,382, respectively, with a total of 20 repair failures (MRs). Without PM, these values ​​were 636, 7478, and 1,063,382, respectively, with a total of 55 MRs. Figure 7 As shown in (b), a reasonable PM strategy can significantly optimize the objective function and greatly reduce the frequency of machine failures. The job agent and machine agent trained by MOSAC can learn reasonable scheduling strategies and respond to different disturbances based on real-time processing status, thereby ensuring the performance of the scheduling scheme.

[0147] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A flexible job shop scheduling method based on preference-driven graph reinforcement learning, characterized in that, include: Establish a multi-objective optimization model for the corresponding flexible operation workshop; Obtain the job set and machine set corresponding to the actual production situation and convert them into a heterogeneous graph; The multi-objective optimization model is decomposed, and a multi-objective Markov decision process is constructed. Upon arrival of a new order, the heterogeneous graph structure is updated, and a decision scheme for dynamic scheduling of the workshop is obtained through the multi-objective Markov decision process.

2. The method according to claim 1, characterized in that, The optimization objectives of the multi-objective optimization model include at least: minimizing the total workpiece delay (TWT) and the maximum workpiece completion time (C). max Minimize processing and maintenance costs; The multi-objective optimization model is expressed as follows: Where min and max represent the minimum and maximum value functions, respectively; at time zero, the flexible work shop has a set containing n jobs, and there are m machines in the flexible work shop; C pk For process O pk The completion time, where p represents the workpiece number and k represents the process number; h j C represents the total number of operations for workpiece j; p Represents the processing cost per unit time; the j-th operation of the i-th job O ij Machine M k Execution, O ij The corresponding processing time is p ijk cost M This indicates the costs of preventative maintenance and minor repairs; D represents the completion time of the last step in the i-th job; i Pr represents the delivery date of the i-th job; i This represents the priority weight of the i-th job.

3. The method according to claim 1, characterized in that, The process of transforming heterogeneous graphs includes: The initial characteristics of machine nodes, operation nodes, and operation-machine (OM) pairs in the heterogeneous graph are defined. The heterogeneous graph model is H = (O, M, C, ε), where O is the process node type, M is the machine node, C is the set of directed edges representing the order constraints between processes, and ε represents the edge type. The d-dimensional time encoding function corresponding to the heterogeneous graph is: d represents the number of dimensions, ω1 represents the frequency component of the first dimension; ω d This represents the frequency component of the d-th dimension, used to control the periodic variation characteristics in different dimensions; The heterogeneous diagram represents the workshop status of the flexible work workshop.

4. The method according to claim 3, characterized in that, The heterogeneous graph characterizes the workshop state of the flexible workshop, including: Based on the meta-path, process nodes are embedded in the heterogeneous graph, and the relationships between process nodes with intermediate nodes are fused. The attention coefficient between two process nodes is: in, This is the first type of linear transformation. This is the second type of linear transformation. Represented as an attention weight vector, used for linearly combining and concatenating features; R represents the real number field; d' o d represents the feature dimension of the process node after linear transformation in the attention mechanism. O This represents the dimension representing the characteristics of the original process node; Indicates process node O ij The original feature vector; This indicates the process node O. ip The original feature vector; The machine nodes in the heterogeneous graph are encoded using a graph attention network; Obtain the global features of the heterogeneous graph.

5. The method according to claim 4, characterized in that, The encoding of machine nodes in the heterogeneous graph using a graph attention network includes: Among them, u k,m,n Represents machine node m k With operation node O mn Attention coefficient between, Represents a machine node. Indicates the operation node. Represents the attention weight vector. d m The dimension representing the node's embedded features; Among them, u kq Represents machine node m k With operation node m q Attention coefficient between, X1 and X2 represent three different linear transformation matrices. Represents machine node m q Features, oc kq This indicates a competition relationship between two machine nodes; Among them, u kk This represents the attention coefficient between the machine node and itself. For u k,m,n ,u kq and u kk Normalization is performed to obtain the corresponding normalized attention coefficients β. k,m,n ,β kq and β kk ; The embedding method of the machine nodes in the heterogeneous graph is as follows: This represents the updated machine node embedding vector, where σ represents the non-linear activation function (such as ReLU or LeakyReLU). Indicates the current machine node m k The original embedding vector, O ij Let m represent the j-th operation node of job i. k Let m represent the k-th machine node. q Indicates with m k Other machine nodes that are in competition, Z2 represents the linear transformation matrix acting on the operating node, and q represents the index of the adjacent machine node. This indicates that there is a competing relationship between machine nodes m. q The original embedding vector.

6. The method according to claim 5, characterized in that, The process of obtaining the global features of the heterogeneous graph includes: in, Represents the global features of a MODFJSP instance, where N represents the number of layers in the graph neural network, and O represents the global features of the instance. u Let M represent the set of all operation nodes in the graph. u This represents the set of all machine nodes in the graph.

7. The method according to claim 1, characterized in that, The decomposition strategy for the multi-objective optimization model uses the workshop state and preference weights as the state space and the selection of the process machine as the action space. The decomposition strategy includes: process strategy. and machine strategy μ(a o |s) represents the operation action a ot The corresponding scalar, μ(a) m |s) represents machine action a mt The corresponding scalar, a ot ∈A o a mt ∈A m A o A represents the motion space of a process. m a represents the machine's motion space. o a represents an action in the set of optional process actions. m θ represents an action from the set of available machines for the current process. o Represents the process strategy network, θ m This represents the machine policy network, where s represents the current state and b represents the current state. o b represents the alternative actions in the action space for the corresponding process selection. m This represents the alternative actions in the corresponding machine's action space, and μ represents the scoring function; The extracted features represent the operation actions. Extracted features representing machine actions Representing the global features of the graph, This represents the operation action extracted by the Nth layer of the graph neural network. h represents the representation of the machine action extracted by the Nth layer of the graph neural network. O-M Representation operation – embedding the interaction relationship between machine pairs.

8. The method according to claim 7, characterized in that, The decomposition of the multi-objective optimization model includes: The multi-objective task is decomposed into three objectives, where, for the maximum completion time, the immediate reward r1 = C is derived. max (t-1)-C max (t); for the total accumulated delay TWT, decompose it into the instant reward. r2 = TWT ave (t-1)-TWT ave (t); for the cumulative cost, decompose it into immediate rewards. r3 = Cost ratio (t-1)-Cost ratio (t); where C max (t) represents the maximum completion time at the current moment, TWT ave (t) represents the average total delay time at the current moment, and Cost is... ratio (t) represents the cost proportion at the current moment; the weighted average of the three objectives is R. t =ω1r1 + ω2r2 + ω3r3, where r1, r2, and r3 represent the immediate rewards of the corresponding sub-goals, and ω1, ω2, and ω3 represent the corresponding weighting coefficients, ω1 + ω 2+ ω3=1.

9. The method according to claim 8, characterized in that, In multi-objective Markov decision-making, the reward vector is normalized as follows: R' t Let R denote the normalized reward vector of the objective at time t. max,l,t R min,l,t Let R be the maximum and minimum values ​​of the l-th objective function value in time interval t, where l = 1, 2, 3. 1,t R 2,t and R 3,t These are the reward values ​​for the first, second, and third objectives at time t, respectively.

10. The method according to claim 1, characterized in that, The process of updating the heterogeneous graph structure also includes: The heterogeneous diagram structure is updated based on the maintenance status of the machines and equipment in the flexible operation workshop.

Images