Semiconductor factory scheduling method, device and equipment based on maximum entropy reinforcement learning

Abstract

The application provides a semiconductor factory scheduling method and device based on maximum entropy reinforcement learning, equipment and medium, belongs to the field of semiconductor scheduling, the method comprises the following steps: acquiring the state information of the current production line; constructing a heterogeneous graph of production scheduling in real time according to the state information; using a graph neural network to analyze the heterogeneous graph to obtain a characteristic vector; generating action embedding of work order node-machine node through the heterogeneous graph Transformer architecture; based on the action embedding, a strategy network of maximum entropy reinforcement learning is used to generate multiple feasible schedules; a specific schedule is selected from the feasible schedule through the strategy network; the heterogeneous graph is adjusted according to the specific schedule, and updated state information is generated; the specific schedule executed is evaluated to generate a reward signal; the strategy network is optimized and adjusted in real time according to the reward signal and the maximum entropy reinforcement learning algorithm. Through the processing scheme of the application, high-quality schemes can be generated within seconds when facing scheduling requirements in actual production, perfectly matching the dynamic adjustment requirements of the semiconductor workshop.

Description

Technical Field

[0001] This application relates to the field of semiconductor scheduling, and in particular to a semiconductor plant scheduling method, apparatus, computer equipment, and storage medium based on maximum entropy reinforcement learning. Background Technology

[0002] Production scheduling, due to its wide range of business applications and NP-hard computational complexity, has always been a core area of ​​algorithm research. Semiconductor shop floor scheduling is a particularly complex flexible job shop problem (FJSP), requiring consideration not only that workpieces can be processed on multiple machines, but also the special constraint of the "vertical machine constraint"—that is, workpieces must be processed continuously by the same set of machines in the lithography process. This constraint makes the model exceptionally complex, making it difficult to quickly find the optimal solution in large-scale production. Current mainstream solutions mainly fall into three categories: exact algorithms represented by mixed-integer programming, metaheuristic algorithms represented by genetic and annealing algorithms, and experience-based rule-based algorithms.

[0003] When dealing with the Full-Time Scheduling Problem (FJSP) in a semiconductor shop floor, precise algorithms often require hours or even days to solve due to complex constraints such as the "vertical machine constraint." This response speed is far from meeting the requirements for production plans that need frequent dynamic adjustments. On the other hand, while metaheuristic and purely rule-based algorithms can quickly provide solutions, their effectiveness relies excessively on historical experience and specific scenarios. When shop floor production parameters or order combinations change, the scheduling quality generated by these algorithms significantly decreases, exposing their shortcomings in generalization and adaptability.

[0004] Therefore, there is an urgent need for a method that can meet the scheduling needs of semiconductor plants using advanced processes. Summary of the Invention

[0005] Therefore, in order to overcome the shortcomings of the prior art, the present invention provides a semiconductor plant scheduling method, apparatus, computer equipment, and storage medium based on maximum entropy reinforcement learning that can respond extremely quickly, make real-time decisions, and generate high-quality solutions within seconds when facing scheduling needs in actual production, perfectly matching the dynamic adjustment requirements of the semiconductor plant.

[0006] To achieve the above objectives, this invention provides a semiconductor plant scheduling method based on maximum entropy reinforcement learning, comprising: acquiring the status information of all work orders and machines on the current production line; constructing a heterogeneous graph of production scheduling in real time based on the status information, wherein the heterogeneous graph includes machine nodes and corresponding work order nodes, where each machine node represents a machine, and each work order node represents a work order that can be processed by the machine node, and the processing relationship between machine nodes and work order nodes is represented by connections; extracting features of each node in the heterogeneous graph using a graph neural network, performing linear mapping and type embedding respectively, and unifying them into a fixed-dimensional feature vector in a multi-dimensional semantic space; and using a multi-head attention mechanism of the heterogeneous graph Transformer architecture to... The heterogeneous graph in the multidimensional semantic space is subjected to relationship perception, and action embeddings connecting work order nodes and machine nodes are generated. Based on the action embeddings, a policy gradient algorithm of maximum entropy reinforcement learning is used to generate multiple feasible schedules, each of which includes at least one scheduling task of a machine node executing a work order node. The multiple feasible schedules are sampled or greedily selected through a policy network to determine a specific schedule. The heterogeneous graph is adjusted according to the specific schedule, and when a specific schedule is detected to be executed, updated state information is obtained. The executed specific schedule is evaluated according to a preset optimization objective and the updated state information, and a reward signal is generated. The policy network is optimized and adjusted in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

[0007] In one embodiment, the step of using a graph neural network to extract features of each node in a heterogeneous graph, performing linear mapping and type embedding respectively, and unifying them into a fixed-dimensional feature vector in a multidimensional semantic space includes: creating independent linear mapping layers for different types of nodes and connections in the heterogeneous graph in the multidimensional semantic space; performing linear mapping on the feature vectors corresponding to the nodes or connections in the heterogeneous graph through the linear mapping layers, and adding the corresponding type identifier during mapping; unifying the dimensions of different types of nodes and arcs in the multidimensional semantic space, and outputting the fixed-dimensional feature vector of the heterogeneous graph in the multidimensional semantic space.

[0008] In one embodiment, the step of constructing a heterogeneous graph for production scheduling based on the status information includes: creating an empty heterogeneous graph; creating work order nodes and machine nodes based on the status information, and adding edges of different types according to different connection relationships between the work order nodes and the machine nodes; and constructing feature vectors for each work order node and each machine node.

[0009] In one embodiment, the step of obtaining updated status information when the specific schedule is detected to be executed includes: when the specific schedule is executed, identifying the scheduling tasks corresponding to the work order node and the machine node; updating the connection status between the corresponding work order node and the machine node in the heterogeneous graph according to the scheduling task, and generating updated status information.

[0010] In one embodiment, the step of sampling or greedily selecting from multiple feasible schedules through a policy network to determine a specific schedule includes: the policy network using a formula... By sampling or greedily selecting from multiple feasible schedules, a specific schedule is determined, wherein... α is the entropy of the policy network; α is the learnable temperature coefficient used to balance the weights of reward and entropy; J(π) is the objective function of the policy network; π is the policy network; t is the time step; E is the expected value, representing the average over all possible outcomes; π(s) t ,a t )~ρ represents the state-action distribution; s t It is the state at time t; a t It is the action taken at time t; ρπ is the trajectory distribution generated by strategy π; r(s) t ,a t H(...) is the reward function; H(...) is the entropy, representing the randomness of the policy network.

[0011] In one embodiment, the heterogeneous graph in the multidimensional semantic space is subjected to relationship perception through the multi-head attention mechanism of the heterogeneous graph Transformer architecture, and action embeddings connecting work order nodes and machine nodes are generated. This includes: perceiving the relationships between various types of nodes and connections in the heterogeneous graph of the multidimensional semantic space through the multi-head attention mechanism of the heterogeneous graph Transformer architecture, and determining the corresponding node types and connection types; for each node or connection, determining the node embedding corresponding to the node or connection using the multi-head attention mechanism; aggregating and processing adjacent node-node or node-connection relationships of different types through the multi-head attention mechanism, and generating action embeddings connecting work order nodes and machine nodes based on the node embeddings.

[0012] In one embodiment, the step of real-time optimization and adjustment of the policy network based on the reward signal and the maximum entropy reinforcement learning algorithm includes: using the formula The reward signal is learned, where J(α) is the objective function for α, logπ(a|s) is the current entropy within the expected value E, -logπ(a|s) represents the actual entropy of the current policy, and H(target) is the target entropy, which is a set hyperparameter and is a constant.

[0013] A semiconductor plant scheduling device based on maximum entropy reinforcement learning, the device comprising: a state awareness module for acquiring state information of all work orders and machines on the current production line; a heterogeneous graph construction module for constructing a heterogeneous graph of production scheduling in real time based on the state information, the heterogeneous graph including machine nodes and corresponding work order nodes, wherein the machine node represents the machine, the work order node represents the work order that can be processed by the machine node, and the processing relationship between machine nodes and work order nodes is represented by connections; a feature mapping module for extracting features of each node in the heterogeneous graph using a graph neural network, performing linear mapping and type embedding respectively, and unifying them into a fixed-dimensional feature vector in a multi-dimensional semantic space; and a heterogeneous graph encoding module for encoding the heterogeneous graph in the multi-dimensional semantic space using a multi-head attention mechanism of the heterogeneous graph Transformer architecture. The system comprises the following modules: a row relationship perception module, which generates action embeddings connecting work order nodes and machine nodes; a policy value network module, which generates multiple feasible schedules based on the action embeddings using a policy gradient algorithm based on maximum entropy reinforcement learning, wherein each feasible schedule includes at least one machine node executing the scheduling task of the work order node; an action sampling constraint module, which samples or greedily selects from the multiple feasible schedules through the policy network to determine a specific schedule; an environment interaction and state transition module, which adjusts the heterogeneous graph according to the specific schedule and obtains updated state information when a specific schedule is detected to be executed; a reward signal generation module, which evaluates the executed specific schedule according to a preset optimization objective and updated state information and generates a reward signal; and a training module, which optimizes and adjusts the policy network in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

[0014] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the above-described method.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0016] Compared with existing technologies, the advantages of this invention are as follows: It employs reinforcement learning to construct an intelligent scheduling model, thereby achieving intelligent construction of specific schedules. Faced with scheduling needs in actual production, it can automatically respond and make real-time decisions, generating high-quality solutions within seconds, perfectly matching the dynamic adjustment requirements of semiconductor workshops. Furthermore, it automatically uses graph neural networks (GNNs) to represent and simulate the entire production process. Through GNNs, elements such as work orders and equipment are abstracted as nodes in a graph, and the complex relationships between nodes (such as work orders and equipment) are deeply learned. Through understanding the topology, the model that runs the decision-making logic possesses unprecedented transferability. When applied to a new workshop, even if the number of work orders and equipment changes significantly, the model can still quickly adapt to the new environment and maintain excellent solution performance by relying on the learned general "node relationships" and "classification patterns." In summary, the method proposed in this application achieves superior scheduling performance while ensuring second-level response efficiency, and can successfully generalize to large-scale and new scenario problems not seen during training, demonstrating strong industrial application potential. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a semiconductor plant scheduling method based on maximum entropy reinforcement learning in one embodiment. Figure 2 This is a flowchart illustrating the steps of extracting feature vectors using a graph neural network in one embodiment. Figure 3 This is a partial schematic diagram illustrating the construction of a heterogeneous graph in one embodiment; Figure 4 This is a flowchart illustrating a semiconductor plant scheduling method based on maximum entropy reinforcement learning in another embodiment. Figure 5 This is a block diagram of a semiconductor plant scheduling device based on maximum entropy reinforcement learning in one embodiment. Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0020] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] It should be noted that the following description covers various aspects of embodiments within the scope of protection of this invention. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0022] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0023] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0024] This application provides a semiconductor plant scheduling method based on maximum entropy reinforcement learning, which can be applied to servers or terminals. Terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable smart devices. Servers can be implemented using independent servers or server clusters composed of multiple servers. Figure 1 As shown, the method is applied to a server as an example, and includes the following steps: Step 101: Obtain the status information of all work orders and machines on the current production line.

[0025] Status information includes information used to make specific scheduling decisions. For example, status information may include the current position, processing stage, and waiting status of all work orders on the entire production line, as well as the working status (busy, idle, faulty) of all machines (equipment). Wafers may pass through the same equipment or process multiple times during the manufacturing process, and status information may also include the current order in which the wafer passes through the same equipment or process.

[0026] The server can directly obtain all work orders and machine status information for the current production line. Alternatively, the server can use an agent to perceive the current production line and obtain all work orders and machine status information. The agent can perceive the current production line based on newly added work order information, a predetermined frequency, or the end signal of a historical schedule. The agent perceives the current semiconductor production line by acquiring environmental information through sensors, data interfaces (APIs), or user input. The agent can be programmed with periodic tasks to periodically acquire environmental information from various input channels. Semiconductor manufacturing involves hundreds or even thousands of steps, each with strict requirements and parameter controls.

[0027] Step 102: Construct a heterogeneous graph of production scheduling in real time based on the status information. The heterogeneous graph includes machine nodes and corresponding work order nodes. Machine nodes represent machines, and work order nodes represent work orders that can be processed at machine nodes. The processing relationship between machine nodes and work order nodes is represented by lines.

[0028] The server constructs a heterogeneous graph for production scheduling based on status information. This graph contains machine nodes and corresponding operation nodes. Machine nodes represent machines, and operation nodes represent operation nodes that can be processed at a machine node. The processing relationships between machine nodes and operation nodes are represented by lines (arcs). Both nodes and lines in the heterogeneous graph have specific features (values) that reflect their status in the current step. Feature vectors can be used in the heterogeneous graph to describe the attributes and characteristics of nodes or edges. Feature vectors can contain various types of data, such as numerical and categorical data, to represent the intrinsic attributes of nodes or edges.

[0029] Step 103: Use graph neural networks to extract features from each node in the heterogeneous graph, perform linear mapping and type embedding respectively, and unify them into a fixed-dimensional feature vector in a multi-dimensional semantic space.

[0030] The characteristics of each node can be represented by the node feature vector of the heterogeneous graph. For example, node features can be its own node name, neighboring nodes, connecting edges, connection time periods, etc. These features can be combined into a vector x = [x1, x2, ..., x...].n In this context, x1 can be the name of the current node, x2 can be a neighboring node, and so on. The server uses Graph Neural Networks (GNNs) to extract features from each node in the heterogeneous graph, performs linear mapping and type embedding on each feature, and unifies them into a fixed-dimensional feature vector in a multi-dimensional semantic space. The GNN updates the features of the current node by aggregating the features of its neighboring nodes, thus obtaining a fixed-dimensional feature vector. For each machine node Mk, its neighboring nodes are a set of work order nodes Nt(Mk), which are a set of work orders that can be produced on this machine. These work order nodes are connected to machine node Mk.

[0031] In one embodiment, the server can also assign different weights to different neighbor nodes through an attention mechanism, thereby more flexibly aggregating neighbor features and obtaining a fixed-dimensional feature vector. For example, the server can use a Graph Attention Network (GAT) to extract features from heterogeneous graphs, transforming graph structure information that reinforcement learning cannot recognize into a fixed-dimensional feature vector that can be used for reinforcement learning. The server can also introduce other fusion methods (e.g., temporal, multi-view, etc.) to assign different weights to different neighbor nodes, thereby more flexibly aggregating neighbor features and obtaining a fixed-dimensional feature vector.

[0032] Step 104: Use the multi-head attention mechanism of the heterogeneous graph Transformer architecture to perform relation perception on the heterogeneous graph in the multidimensional semantic space and generate action embeddings that connect work order nodes and machine nodes.

[0033] The server uses a multi-head attention mechanism within the heterogeneous graph architecture's heterogeneous graph structure to perceive relationships within the multi-dimensional semantic space. It determines the node embeddings corresponding to nodes or connections and generates action embeddings that connect work order nodes to machine nodes based on these embeddings. The heterogeneous graph Transformer architecture is a deep learning model specifically designed for processing heterogeneous graph data. It combines the attention mechanism of the Transformer architecture with the advantages of graph neural networks, effectively modeling complex node and edge relationships in heterogeneous graphs. The core idea of ​​this architecture is to solve the heterogeneity problem by learning shared representations that are independent of task and robot form through a scalable Transformer backbone.

[0034] The server uses an independent attention mechanism for each relationship type (work order node, machine node, connection between work order node and machine node, etc.) and assigns different attention scores to different types of work order node-machine node connections. Then, the server aggregates neighbors of different relationship types for a node or connection using a multi-head attention mechanism (for a work order node, neighbors can be work order nodes, machine nodes, connections, etc.), calculates the corresponding attention, and merges them according to weighted aggregation to obtain the node embedding for that node or connection. The server also aggregates adjacent node-node or node-connection connections of different relationship types using a multi-head attention mechanism and generates an action embedding that connects the work order node and machine node based on this node embedding.

[0035] Step 105: Based on action embedding, use the maximum entropy reinforcement learning policy gradient algorithm to generate multiple feasible schedules. The feasible schedules include at least the scheduling tasks of machine nodes executing work order nodes.

[0036] A feasible schedule must include at least the scheduled tasks for machine nodes to execute work order nodes. The scheduled tasks in a feasible schedule are those that machine nodes can theoretically execute work order nodes, but these scheduled tasks may not be the optimal task allocation combination.

[0037] The server, based on action embedding, uses the Soft Actor-Critic (SAC) algorithm, a maximum entropy reinforcement learning algorithm, to generate multiple feasible schedules. Each feasible schedule includes at least one task where machine nodes execute work order nodes. The core idea of ​​the maximum entropy reinforcement learning algorithm is to maximize the policy entropy while maximizing the expected reward, thus promoting exploration. Discrete SAC (Discrete Action Space SAC) is an improvement on the original algorithm, allowing it to directly handle discrete actions. SAC balances exploration and exploitation by optimizing the following objectives: in: It is the entropy of the strategy; It is a learnable temperature coefficient used to balance the weights of "reward" and "entropy," adaptively adjusting the weight of the entropy term to balance optimality and exploration.

[0038] Step 106: Sample or greedily select from multiple feasible schedules through a policy network to determine the specific schedule.

[0039] The server analyzes feature vectors based on the real-time trained policy network (Actors) and generates specific schedules. These schedules at least include the execution of work order node scheduling tasks by machine nodes. Scheduling tasks associate machine nodes and work order nodes; for example, a scheduling task might assign a work order X from the waiting queue to a newly idle machine Y for processing. A specific schedule can contain multiple scheduling tasks. The server can sample or greedily select feasible "work order-machine" pairs based on the Actor distribution and use a masking mechanism to block actions that do not meet process or resource constraints. Sampling and greedy selection are two different analysis methods.

[0040] The sampling selection method samples from the probability distribution behind the mask. The greedy selection method selects the effective action with the highest probability.

[0041] Step 107: Adjust the heterogeneous graph according to the specific schedule, and obtain the updated status information when the specific schedule is detected to be executed.

[0042] The server adjusts the heterogeneous graph based on the specific schedule and obtains updated status information when a specific schedule is detected to be executed. The server updates the status graph according to the selected "work order-machine" combination, changing the corresponding connection from dashed lines to solid lines and removing other dashed lines, driving the system to the new state. When a specific schedule is detected to be executed, the server obtains updated status information. The server can change the connections between work order nodes and machine nodes in the heterogeneous graph according to the specific schedule; for example, it can adjust the connection between the selected work order and the matching machine to a solid line and remove the dashed lines between the work order and other machines.

[0043] Step 108: Evaluate the specific schedule to be executed based on the preset optimization goals and updated status information, and generate a reward signal.

[0044] The server evaluates the executed schedule based on preset optimization goals and updated status information, and generates a reward signal. The optimization goal can be the expected completion time of the scheduled task or the actual completion time of the most recent scheduled task. The server obtains the current completion time of the executed schedule based on the updated status information, and evaluates the executed schedule based on the current completion time and the optimization goal: the server can evaluate the executed schedule by calculating the difference between the current completion time and the optimization goal; the server can also evaluate the executed schedule by calculating the ratio between the current completion time and the optimization goal. The server generates a reward signal based on the evaluation result.

[0045] Step 109: Optimize and adjust the policy network in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

[0046] The server optimizes and adjusts the policy network in real time based on the reward signal and the maximum entropy reinforcement learning algorithm. The server receives the reward signal and inputs the specific schedule into the pre-trained algorithm or model. Then, the agent iteratively updates the policy network.

[0047] Thus, the entire scheduling and training feedback process forms a Markov Decision Process (MDP). A standard MDP consists of five core elements: State, Action, Transition, Reward, and Policy. The decision-making process of an MDP is as follows: Figure 4 As shown, the decision-making process of MDP iterates at discrete decision epochs, and all MDPs are executed by agents. A decision epoch refers to the agent's renewed awareness of the current production line. For example, the trigger for a decision epoch t could be the initial moment of the system, or the completion of a previous scheduling action causing a device to enter an idle state.

[0048] For each decision point t, the agent's decision-making process is as follows: State Observation: The agent first comprehensively perceives the current system state St. This state contains all the information needed for decision-making, such as the current location, processing stage, and waiting status of all work orders on the entire production line, as well as the working status (busy, idle, faulty) of all equipment.

[0049] Action Selection: Based on the current state s t The agent selects an optimal action a based on its internal policy π. t The strategy π here is the decision logic learned by the model after training. A specific "action" is a scheduling task, such as: "assigning work order X in the waiting queue to the newly idle device Y for processing".

[0050] State Transition: When action a... t After execution, the system will evolve into a new state s according to its inherent dynamic rules (i.e., state transition probabilities P). t+1 For example, when work order X enters the processing area of ​​equipment Y from the waiting area, the status of equipment Y changes from "idle" to "busy".

[0051] Reward Feedback: The system will reward the action 'a' that was just performed based on preset optimization goals (such as minimizing total completion time, maximizing equipment utilization, etc.). t Evaluate and provide a quantified reward signal r. t (Reward). A good decision will result in a positive reward, while a bad decision will result in a negative reward or penalty. In one embodiment, the reward signal r t The difference in time taken when two states are finally completed is given by the formula r(s). t , a t , s t+1 )=C max (s t+1 )- C max (s t ), where C max The time for the completion of the final process includes both the processing time and the transition time for state transitions.

[0052] Policy Improvement: Reward Signal r t Feedback is fed back to the agent to adjust and optimize its internal policy π. By continuously repeating the "trial-feedback-learning" cycle, the probability of making efficient decisions in policy π continuously increases, while the probability of decisions leading to adverse consequences decreases. The entire MDP process starts from the initial state and iterates according to the above steps until all processes are scheduled and the system reaches the termination state. Through this series of ingenious interactive learning, the model trained by the agent can master the ability to make efficient scheduling decisions in complex and ever-changing production environments.

[0053] The policy network can be learned by a server or agent through training on small batches of historical data. Historical data can include historical state information, historical scheduling information, and historical efficiency feedback. The decision logic can be represented by a trained model or by the probability distribution of different scheduling tasks. The server can learn and adjust the decision logic in real time based on feedback information, enabling autonomous learning and solving larger-scale and more complex scheduling problems, thus lowering the implementation threshold of the entire process.

[0054] The aforementioned method employs data-driven training, eliminating reliance on experience. The entire model training process does not depend on complex production line knowledge or expert experience. Initial training is completed using small batches of data, after which the intelligent agent autonomously learns and solves larger-scale, more complex scheduling problems, lowering the implementation threshold. Facing scheduling needs in actual production, it can generate high-quality solutions within seconds, perfectly matching the dynamic adjustment requirements of semiconductor workshops. When applied to new workshops, even with significant changes in work orders and equipment numbers, the model can quickly adapt to the new environment and maintain excellent solution performance by leveraging its learned general "node relationships" and "classification patterns." This method achieves superior scheduling performance while maintaining second-level response efficiency and successfully generalizes to large-scale and new scenario problems not encountered during training, demonstrating strong industrial application potential. This method uses reinforcement learning to construct an intelligent scheduling model, thereby intelligently constructing specific schedules. Facing scheduling needs in actual production, it can respond rapidly and make real-time decisions, generating high-quality solutions within seconds, perfectly matching the dynamic adjustment requirements of semiconductor workshops. Furthermore, it innovatively uses Graph Neural Networks (GNNs) to represent and simulate the entire production process. GNNs abstract elements such as work orders and equipment into nodes in a graph, and deeply learn the complex relationships between nodes (e.g., work orders and equipment). Through understanding the topological structure, the model for executing decision-making logic possesses unprecedented transferability. When applied to a new workshop, even with significant changes in the number of work orders and equipment, the model can still quickly adapt to the new environment and maintain excellent solution performance by relying on the learned general "node relationships" and "classification patterns." In summary, the method proposed in this application achieves superior scheduling performance while ensuring second-level response efficiency, and can successfully generalize to large-scale and novel scenarios not seen during training, demonstrating strong potential for industrial applications.

[0055] In one embodiment, such as Figure 2 As shown, graph neural networks are used to extract features from each node in a heterogeneous graph, and linear mapping and type embedding are performed respectively to unify them into a fixed-dimensional feature vector in a multi-dimensional semantic space. The steps include: Step 201: The features of the machine node are processed into a machine feature matrix using a first linear transformation, and the features of the work order node are processed into a work order feature matrix using a second linear transformation.

[0056] The features of each machine node and the features of the work order node, such as available time, processing time, and product color, are encoded into one-dimensional vector features. For example, the features of a machine node can be represented by a vector v. k Let k represent the k-th machine node, and the features of the work order node can be represented by the vector u. i Let i represent the i-th work order node. The server uses the first linear transformation W. M The characteristics v of the machine nodek It is processed into a machine feature matrix and then subjected to a second linear transformation W. O The characteristics of the work order node u i Process it into a work order feature matrix.

[0057] Linear transformations can be achieved through matrix W M Or W O This is achieved by... The size of the matrix depends on the dimension of the eigenvectors. When the eigenvectors are n-dimensional, then the linear transformation matrix W... M Or W O It is an m × n matrix, where m is the dimension of the transformed characteristic matrix. Matrix W M Or W O The elements within can be randomly generated by the server or pre-set. The server multiplies the feature vectors by the linear transformation matrix to obtain the feature matrix.

[0058] Step 202: Concatenate the machine feature matrix and the work order feature matrix, and input the resulting concatenated matrix into the multilayer perceptron to calculate the attention coefficient between each machine node and the work order node.

[0059] The server directly concatenates the machine feature matrix and the work order feature matrix. Since both the machine feature matrix and the work order feature matrix are m-dimensional, the resulting concatenated matrix has a dimension of 2m. The server then inputs this concatenated matrix into a multilayer perceptron (MLP) to calculate the attention coefficient e between each machine node and the work order node. ik Multilayer perceptrons are common feedforward neural networks, and the server can select different multilayer perceptrons based on accuracy requirements. In one embodiment, the attention coefficient e ik The specific calculation formula is as follows: e ik = LeakyReLU(a T [W M v k ||W O u i ]), where a is the weight of the MLP, || represents the vector concatenation operation, and LeakyReLU is the function name.

[0060] In one embodiment, the server can also use the softmax function to apply the attention coefficients e to all attention coefficients. ik Normalization is performed to obtain the normalized attention coefficient α. ik This simplifies the subsequent calculations.

[0061] Step 203: The adjacency matrix is ​​used to represent the connection relationship of the heterogeneous graph, and the feature vector corresponding to each machine node is extracted. The elements of the feature vector are obtained by weighted summation of the features and attention coefficients of the work order node and then adding the features of the machine node.

[0062] The server uses an adjacency matrix to represent the connectivity of a heterogeneous graph. Taking a heterogeneous graph G = (V, E) as an example, V is the set of nodes and E is the set of edges. The graph has n nodes, and the adjacency matrix A is an n × n matrix, where A... ij This represents the connection between node i and node j. The elements of the feature vector are obtained by weighted summing of the features and attention coefficients of the work order node, then adding the features of the machine node. The server can extract the feature vector v corresponding to each machine node. k And the feature vector u of the corresponding neighboring node (the work order node generated on this machine). i Then the server will sort the neighboring nodes according to their attention coefficients (e... ik Or α ik The weighted summation, combined with the machine node's own features, yields a feature vector v of the machine node that incorporates the features of its neighboring work order nodes. k This yields the feature representation of the entire graph, resulting in a feature vector. The attention coefficient α is used as the basis for this representation. ik For example, the specific calculation formula is as follows: , where σ represents the activation function, which can be the Sigmoid function or the ReLU function.

[0063] The above method extracts features from heterogeneous graphs through graph attention networks, transforming graph structure information that reinforcement learning cannot recognize into fixed-dimensional feature vectors that can be used for reinforcement learning, thereby enhancing the neural network construction for subsequent agent learning.

[0064] In one embodiment, a graph neural network is used to extract features of each node in a heterogeneous graph, perform linear mapping and type embedding respectively, and unify them into a fixed-dimensional feature vector in a multidimensional semantic space. The steps include: creating independent linear mapping layers in the multidimensional semantic space for different types of nodes and connections in the heterogeneous graph; performing linear mapping on the feature vectors corresponding to nodes or connections in the heterogeneous graph through the linear mapping layers, and adding the corresponding type identifier during mapping; unifying the dimensions of different types of nodes and arcs in the multidimensional semantic space, and outputting the fixed-dimensional feature vector of the heterogeneous graph in the multidimensional semantic space.

[0065] The server creates independent linear mapping layers in the multidimensional semantic space for different types of nodes and connections in the heterogeneous graph. The server uses these linear mapping layers to linearly map the feature vectors corresponding to the nodes or connections in the heterogeneous graph, adding the corresponding type identifier during the mapping process. The server unifies the dimensions of different types of nodes and arcs in the multidimensional semantic space and outputs a fixed-dimensional feature vector of the heterogeneous graph in the multidimensional semantic space.

[0066] In one embodiment, constructing a heterogeneous graph for production scheduling based on status information includes the following steps: creating an empty heterogeneous graph; creating work order nodes and machine nodes based on status information, and adding edges of different types according to different connection relationships between work order nodes and machine nodes; and constructing feature vectors for each work order node and machine node.

[0067] The server creates an empty heterogeneous graph G = (V, E, T, R), where V is the set of nodes, E is the set of edges, T is the set of node types, and R is the set of edge types. In the heterogeneous graph, nodes and edges can have different types. Each type of node and edge has its unique characteristics and properties. Creating an empty heterogeneous graph first provides a clear framework for gradually adding and organizing these different types of data. Creating an empty heterogeneous graph also offers high flexibility, allowing the graph to be built incrementally rather than loading all the data at once, which is beneficial for handling large-scale datasets.

[0068] The server creates work order nodes and machine nodes based on status information, and adds different types of edges based on the different connection relationships between work order nodes and machine nodes. The server can first add different types of nodes to the graph based on the collected status information. For example, the server can create a work order node for each work order and set its feature vector, including the work order ID, name, estimated start time, and estimated end time. Then, the server adds different types of edges based on the status information to represent the relationships between nodes. Figure 3 As shown, dashed lines connecting work orders and machines represent optional processing relationships, while solid lines represent already scheduled processing relationships. The server can also use pre-defined logic in an empty heterogeneous graph to verify whether the data meets expectations and determine the rationality of the graph structure when adding nodes or edges.

[0069] The server constructs feature vectors for each work order node and machine node. The feature vectors of different nodes can contain different content. For example, the feature vector of a work order node can contain the work order ID, name, estimated start time, estimated end time, required resources, etc.; the feature vector of a machine node can contain the machine ID, name, available time, assigned work order, etc.

[0070] The above method constructs a heterogeneous graph based on the state information of the scheduling problem. This graph can be used for further analysis to facilitate the optimization of task scheduling.

[0071] In one embodiment, when a specific schedule is detected to be executed, the agent obtains updated state information, including: when a specific schedule is executed, identifying the scheduling tasks corresponding to the work order node and the machine node; updating the connection state between the corresponding work order node and the machine node in the heterogeneous graph according to the scheduling task, and generating updated state information.

[0072] When a specific schedule is executed, the server identifies the scheduling task corresponding to the work order node and the machine node. Based on the scheduling task, the server updates the connection status between the corresponding work order node and machine node in the heterogeneous graph, generating updated status information. For example, the connection between the selected work order and the matching machine is changed to a solid line, and the dashed lines between the work order and other machines are removed.

[0073] The updated status information can include the connection information between the selected work order and the matching machine, as well as information such as the execution time of the work order on the machine.

[0074] The above method updates the connection status between work order nodes and machine nodes in the heterogeneous graph in a timely manner, thereby facilitating subsequent server classification training.

[0075] In one embodiment, a specific schedule is determined by sampling or greedily selecting from multiple feasible schedules through a policy network, including: the policy network using a formula... By sampling or greedily selecting from multiple feasible schedules, a specific schedule is determined, wherein... α is the entropy of the policy network; α is the learnable temperature coefficient used to balance the weights of reward and entropy; J(π) is the objective function of the policy network; π is the policy network; t is the time step; E is the expected value, representing the average over all possible outcomes; π(s) t ,a t )~ρ represents the state-action distribution; s t It is the state at time t; a t It is the action taken at time t; ρπ is the trajectory distribution generated by strategy π; r(s) t ,a t H(...) is the reward function; H(...) is the entropy, representing the randomness of the policy network.

[0076] α is the entropy of the policy network; α is a learnable temperature coefficient used to balance the weights of reward and entropy.

[0077] J(π) is the objective function of the policy. This is the total value that the algorithm attempts to maximize. It includes not only the traditional reward but also the randomness (entropy) of the policy.

[0078] π stands for Policy, which can be a function or a neural network responsible for deciding what action to take based on the current state.

[0079] T stands for Time Step, which represents each step in the decision-making process (e.g., the time of each task assignment in scheduling).

[0080] E stands for Expectation, which represents the average of all possible outcomes.

[0081] π(s t ,a t )~ρ represents the state-action distribution.

[0082] s t (State) is the state at time t (e.g., current machine load, state of workpieces waiting to be processed).

[0083] a t (Action) is the action taken at time t (e.g., choosing which workpiece to process).

[0084] ρπ is the trajectory distribution generated by policy π, which represents the probability of accessing a certain combination of state and action under the current policy.

[0085] r(s t ,a t The reward function is the feedback provided by the environment. Good performance (e.g., high scheduling efficiency, low delays) earns positive points, while poor performance earns negative points.

[0086] α (Alpha) is the temperature coefficient, a weighting parameter. α is used to balance the pursuit of high rewards with the maintenance of exploratory behavior (high entropy). The larger α is, the more the algorithm tends to explore (more random actions); the smaller α is, the more the algorithm tends to utilize known high-scoring actions (more deterministic actions).

[0087] H(…) is entropy, which represents the degree of randomness of the strategy.

[0088] In the discrete space, actions do not need to be sampled; the expected Actor (policy) loss can be directly calculated by summation. , L πThis represents the loss value of the policy network, which the neural network updates its parameters by minimizing.

[0089] Es~D means that a batch of states s are randomly selected from the training set D for calculation.

[0090] ∑a is the expectation calculated by summing all possible actions a.

[0091] π(a|s) is the probability output by the policy network, that is, the probability of choosing action a in state s.

[0092] αlogπ(a|s) is the entropy regularization term, used to encourage the strategy to maintain randomness (exploration).

[0093] min i Qθ i (s,a) is a dual-critic mechanism. SAC uses two critic networks (Q1 and Q2) to estimate the value of an action. Taking the minimum of the two (min) is to prevent overestimation bias, thus making the algorithm more robust.

[0094] The goal of policy networks is to find an action distribution that maximizes the combined effect of entropy (exploration) and Q-value (reward).

[0095] Temperature parameter (α) loss .

[0096] Critic (Q-function) loss

[0097] Where LQ is the loss value (mean squared error, MSE) of the Critic network.

[0098] (s,a,r,s)~D is a quadruple sampled from the training set: current state, action, reward, and next state.

[0099] Qθ(s,a) is the value predicted by the current Critic network.

[0100] Y is the target value, which can be expressed by a formula: , Where r is the immediate reward, which is the score obtained immediately by the current scheduled operation.

[0101] γ (Gamma) is a discount factor used to discount future rewards in the current scheduling operation analysis (usually a value no greater than 1, for example, γ can be 0.99), indicating that the further into the future, the more uncertain the future.

[0102] s′ represents the state at the next moment.

[0103] ∑a′ is the expectation of all possible actions at the next time step (which is also a characteristic of discrete SAC).

[0104] Q θi These are the parameters of the target network.

[0105] -αlogπ(a′∣s′) is the future entropy reward. When calculating the target value, we not only look at how many points can be obtained in the future (Q value), but also how much room for choice is retained in the future (entropy).

[0106] The above method trains the policy network using the maximum entropy reinforcement learning algorithm and obtains a policy probability distribution, such that the selection probability is proportional to the corresponding Q-value function, thereby achieving policy learning and optimization.

[0107] In one embodiment, such as Figure 5 As shown, a semiconductor plant scheduling device based on maximum entropy reinforcement learning is provided. The device includes a state perception module 501, a heterogeneous graph construction module 502, a feature mapping module 503, a heterogeneous graph encoding module 504, a policy value network module 505, an action sampling constraint module 506, an environment interaction and state transition module 507, a reward signal generation module 508, and a training module 509.

[0108] The status perception module 501 is used to use an intelligent agent to perceive the current production line and obtain the status information of all work orders and machines on the current production line.

[0109] The heterogeneous graph construction module 502 is used to construct a heterogeneous graph of production scheduling in real time based on status information. The heterogeneous graph includes machine nodes and corresponding work order nodes. Machine nodes represent machines, and work order nodes represent work orders that can be processed at machine nodes. The processing relationship is represented by the connection between machine nodes and work order nodes.

[0110] The feature mapping module 503 is used to extract the features of each node in the heterogeneous graph using a graph neural network, perform linear mapping and type embedding respectively, and unify them into a fixed-dimensional feature vector in a multi-dimensional semantic space.

[0111] The heterogeneous graph encoding module 504 is used to perform relation perception on the heterogeneous graph in the multidimensional semantic space through the multi-head attention mechanism of the heterogeneous graph Transformer architecture, and generate action embeddings that connect work order nodes and machine nodes.

[0112] The policy value network module 505 is used to generate multiple feasible schedules based on action embedding and using the policy gradient algorithm of maximum entropy reinforcement learning. The feasible schedules include at least the scheduling tasks of the work order nodes executed by the machine nodes.

[0113] The action sampling constraint module 506 is used to sample or greedily select from multiple feasible schedules through a policy network to determine the specific schedule.

[0114] The environment interaction and state transition module 507 is used to adjust the heterogeneous graph according to the specific schedule, and to obtain updated state information when the specific schedule is detected to be executed.

[0115] The reward signal generation module 508 is used to evaluate the specific schedule being executed based on preset optimization goals and updated status information, and to generate reward signals.

[0116] Training module 509 is used to optimize and adjust the policy network in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

[0117] In one embodiment, the feature vector extraction module includes: The linear transformation unit is used to process the features of the machine node into a machine feature matrix using a first linear transformation, and to process the features of the work order node into a work order feature matrix using a second linear transformation.

[0118] The attention coefficient calculation unit is used to concatenate the machine feature matrix and the work order feature matrix, and input the resulting concatenated matrix into the multilayer perceptron to calculate the attention coefficient between each machine node and the work order node.

[0119] The vectorization unit is used to represent the connection relationship of the heterogeneous graph using an adjacency matrix and extract the feature vector corresponding to each machine node. The elements of the feature vector are obtained by weighted summation of the features and attention coefficients of the work order node and then adding the features of the machine node.

[0120] In one embodiment, the heterogeneous graph construction module includes: The empty graph creation unit is used to create an empty heterogeneous graph.

[0121] The element creation unit is used to create work order nodes and machine nodes based on status information, and to add edges of different types according to the different connection relationships between work order nodes and machine nodes.

[0122] The vector construction unit is used to construct feature vectors for each work order node and machine node.

[0123] In one embodiment, the update information acquisition module includes: The scheduling task identification unit is used to identify the scheduling task corresponding to the work order node and the machine node when a specific schedule is executed. The update unit is used to update the connection status between the corresponding work order node and machine node in the heterogeneous graph according to the scheduling task, and generate update status information.

[0124] Specific limitations regarding the semiconductor plant scheduling apparatus based on maximum entropy reinforcement learning can be found in the limitations of the semiconductor plant scheduling method based on maximum entropy reinforcement learning mentioned above, and will not be repeated here. Each module in the aforementioned semiconductor plant scheduling apparatus based on maximum entropy reinforcement learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0125] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as decision logic. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a semiconductor plant scheduling method based on maximum entropy reinforcement learning.

[0126] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: using an agent to perceive the current production line and obtain the status information of all work orders and machines on the current production line; constructing a heterogeneous graph of production scheduling based on the status information, the heterogeneous graph containing machine nodes and corresponding work order nodes, where machine nodes represent machines, work order nodes represent work orders that can be processed at machine nodes, and processing relationships are represented by connections between machine nodes and work order nodes; using a graph neural network to extract features of each node in the heterogeneous graph and encode them into fixed-dimensional feature vectors; using an agent to analyze the feature vectors based on real-time trained decision logic and generate a specific schedule, the specific schedule including at least the scheduling tasks of machine nodes executing work order nodes.

[0128] In one embodiment, when the processor executes a computer program, it implements the use of a graph neural network to extract features of each node in a heterogeneous graph and encodes them into a fixed-dimensional feature vector. This includes: acquiring the status information of all work orders and machines on the current production line; constructing a heterogeneous graph for production scheduling in real time based on the status information, where the graph includes machine nodes and corresponding work order nodes, where machine nodes represent machines and work order nodes represent work orders that can be processed at a machine node, and connections between machine nodes and work order nodes represent processing relationships; extracting features of each node in the heterogeneous graph using a graph neural network, performing linear mapping and type embedding respectively, and unifying them into a fixed-dimensional feature vector in a multi-dimensional semantic space; and transforming the heterogeneous graph. The multi-head attention mechanism of the ER architecture enables relation perception in the heterogeneous graph of the multi-dimensional semantic space and generates action embeddings that connect work order nodes and machine nodes. Based on the action embeddings, a policy gradient algorithm using maximum entropy reinforcement learning is used to generate multiple feasible schedules. Each feasible schedule includes at least machine nodes executing the scheduling tasks of work order nodes. The policy network samples or greedily selects from the multiple feasible schedules to determine the specific schedule. The heterogeneous graph is adjusted according to the specific schedule, and when a specific schedule is detected to be executed, the updated state information is obtained. The executed specific schedule is evaluated according to the preset optimization objective and the updated state information, and a reward signal is generated. The policy network is optimized and adjusted in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

[0129] In one embodiment, the heterogeneous graph for constructing production scheduling based on state information, implemented by the processor when executing a computer program, includes: creating an empty heterogeneous graph; creating work order nodes and machine nodes based on state information, and adding edges of different types according to different connection relationships between work order nodes and machine nodes; and constructing feature vectors for each work order node and machine node.

[0130] In one embodiment, when the processor executes a computer program, it implements the following mechanism: when a specific schedule is detected to be executed, the agent obtains updated state information, including: when a specific schedule is executed, the agent identifies the scheduling task corresponding to the work order node and the machine node; the agent updates the connection state between the corresponding work order node and the machine node in the heterogeneous graph according to the scheduling task, and generates updated state information.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: acquiring the status information of all work orders and machines on the current production line; constructing a heterogeneous graph of production scheduling in real time based on the status information, the heterogeneous graph containing machine nodes and corresponding work order nodes, where machine nodes represent machines, work order nodes represent work orders that can be processed at machine nodes, and processing relationships are represented by connections between machine nodes and work order nodes; extracting features of each node in the heterogeneous graph using a graph neural network, performing linear mapping and type embedding respectively, and unifying them into a fixed-dimensional feature vector in a multi-dimensional semantic space; and using a heterogeneous graph Transformer framework. The multi-head attention mechanism of the architecture enables relation perception in the heterogeneous graph of the multi-dimensional semantic space and generates action embeddings that connect work order nodes and machine nodes. Based on the action embeddings, a policy gradient algorithm of maximum entropy reinforcement learning is used to generate multiple feasible schedules, each of which includes at least machine nodes executing the scheduling tasks of work order nodes. The policy network samples or greedily selects from the multiple feasible schedules to determine the specific schedule. The heterogeneous graph is adjusted according to the specific schedule, and when a specific schedule is detected to be executed, the updated state information is obtained. The executed specific schedule is evaluated according to the preset optimization objective and the updated state information, and a reward signal is generated. The policy network is optimized and adjusted in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

[0132] In one embodiment, when the computer program is executed by the processor, it implements the extraction of features of each node in the heterogeneous graph using a graph neural network and encodes them into a fixed-dimensional feature vector. This includes: processing the features of machine nodes into a machine feature matrix using a first linear transformation and processing the features of work order nodes into a work order feature matrix using a second linear transformation; concatenating the machine feature matrix and the work order feature matrix and inputting the resulting concatenated matrix into a multilayer perceptron to calculate the attention coefficient between each machine node and the work order node; using an adjacency matrix to represent the connection relationships in the heterogeneous graph and extracting the feature vector corresponding to each machine node, wherein the elements of the feature vector are obtained by weighted summation of the features of the work order node and the attention coefficient, plus the features of the machine node.

[0133] In one embodiment, the heterogeneous graph for constructing production scheduling based on state information, implemented by the computer program when executed by the processor, includes: creating an empty heterogeneous graph; creating work order nodes and machine nodes based on the state information, and adding edges of different types according to different connection relationships between work order nodes and machine nodes; and constructing feature vectors for each work order node and machine node.

[0134] In one embodiment, when the computer program is executed by the processor, the agent obtains updated state information when a specific schedule is detected to be executed, including: when the specific schedule is executed, the agent identifies the scheduling task corresponding to the work order node and the machine node; the agent updates the connection state between the corresponding work order node and the machine node in the heterogeneous graph according to the scheduling task, and generates updated state information.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A semiconductor plant scheduling method based on maximum entropy reinforcement learning, characterized in that, include: Obtain the status information of all work orders and machines on the current production line; Based on the status information, a heterogeneous graph of production scheduling is constructed in real time. The heterogeneous graph includes machine nodes and corresponding work order nodes. The machine node represents the machine, and the work order node represents the work order that can be processed by the machine node. The processing relationship between the machine node and the work order node is represented by the connection. Graph neural networks are used to extract features from nodes in heterogeneous graphs, and linear mapping and type embedding are performed respectively, which are then unified into fixed-dimensional feature vectors in a multi-dimensional semantic space. The heterogeneous graph in the multidimensional semantic space is subjected to relation perception through the multi-head attention mechanism of the heterogeneous graph Transformer architecture, and action embeddings that connect work order nodes and machine nodes are generated. Based on the action embedding, a policy gradient algorithm using maximum entropy reinforcement learning is used to generate multiple feasible schedules, and the feasible schedules include at least the scheduling tasks of machine nodes executing work order nodes. The policy network samples or greedily selects from multiple feasible schedules to determine the specific schedule in seconds. The heterogeneous graph is adjusted according to the specific schedule, and when the specific schedule is detected to be executed, the updated status information is obtained. The specific schedule to be executed is evaluated based on the preset optimization goals and updated status information, and a reward signal is generated. The policy network is optimized and adjusted in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

2. The semiconductor plant scheduling method according to claim 1, characterized in that, The process involves using a graph neural network to extract features from each node in a heterogeneous graph, performing linear mapping and type embedding respectively, and unifying them into a fixed-dimensional feature vector in a multi-dimensional semantic space, including: For different types of nodes and connections in heterogeneous graphs, separate linear mapping layers are created in the multidimensional semantic space respectively; The linear mapping layer performs linear mapping on the feature vectors corresponding to nodes or connections in the heterogeneous graph, and adds the corresponding type identifier during the mapping process. Unify the dimensions of different types of nodes and arcs in the multidimensional semantic space, and output the feature vector of the heterogeneous graph with fixed dimensions in the multidimensional semantic space.

3. The semiconductor plant scheduling method according to claim 1, characterized in that, The step of constructing a heterogeneous graph of production scheduling based on the status information includes: Create an empty heterogeneous graph; Based on the status information, create work order nodes and machine nodes, and add edges of different types according to the different connection relationships between the work order nodes and the machine nodes; A feature vector is constructed for each work order node and each machine node.

4. The semiconductor plant scheduling method according to claim 1, characterized in that, When the execution of the specific schedule is detected, the updated status information is obtained, including: When the specific schedule is executed, the scheduling tasks corresponding to the work order node and the machine node are identified; According to the scheduling task, the connection status between the corresponding work order node and the machine node in the heterogeneous graph is updated to generate updated status information.

5. The semiconductor plant scheduling method according to claim 1, characterized in that, The step of sampling or greedily selecting from multiple feasible schedules through a policy network to determine a specific schedule includes: The policy network uses the formula By sampling or greedily selecting from multiple feasible schedules, a specific schedule is determined. in, α is the entropy of the policy network; α is a learnable temperature coefficient used to balance the weights of reward and entropy. J(π) is the objective function of the policy network; π is the policy network; t is the time step; E is the expected value, which represents the average value over all possible outcomes. π(s t ,a t ) ρ represents the state-action distribution; s t It is the state at time t; a t It is the action taken at time t; ρπ is the trajectory distribution generated by strategy π; r(s t ,a t ) is the reward function; H(…) is entropy, representing the degree of randomness in the policy network.

6. The semiconductor plant scheduling method according to claim 1, characterized in that, The heterogeneous graph in the multi-dimensional semantic space is subjected to relationship awareness through a multi-head attention mechanism of the heterogeneous graph Transformer architecture, and action embeddings connecting work order nodes and machine nodes are generated, including: The multi-head attention mechanism of the heterogeneous graph Transformer architecture is used to perceive the relationships between various types of nodes and connections in the heterogeneous graph of the multidimensional semantic space, and to determine the corresponding node types and connection types. For each node or connection, the multi-head attention mechanism of the heterogeneous graph Transformer architecture is used to determine the node embedding corresponding to the node or connection. The multi-head attention mechanism aggregates adjacent nodes of different relationship types or nodes-connections, and generates action embeddings that connect work order nodes and machine nodes based on the node embedding.

7. The semiconductor plant scheduling method according to claim 1, characterized in that, The step of optimizing and adjusting the policy network in real time based on the reward signal and the maximum entropy reinforcement learning algorithm includes: Using formula Learn from reward signals. Where J(α) is the objective function for α. logπ(a|s) is the current entropy within the mathematical expectation E. logπ(a|s) represents the actual entropy of the current policy; H(target) is the target entropy, a set hyperparameter that is a constant.

8. A semiconductor plant scheduling device based on maximum entropy reinforcement learning, characterized in that, The device includes: The status awareness module is used to obtain the status information of all work orders and machines on the current production line; The heterogeneous graph construction module is used to construct a heterogeneous graph of production scheduling in real time based on the status information. The heterogeneous graph includes machine nodes and corresponding work order nodes. The machine node represents the machine, and the work order node represents the work order that can be processed by the machine node. The processing relationship is represented by the connection between the machine node and the work order node. The feature mapping module is used to extract the features of each node in the heterogeneous graph using graph neural networks, perform linear mapping and type embedding respectively, and unify them into a fixed-dimensional feature vector in a multi-dimensional semantic space. The heterogeneous graph encoding module is used to perform relation perception on the heterogeneous graph in the multidimensional semantic space through the multi-head attention mechanism of the heterogeneous graph Transformer architecture, and generate action embeddings that connect work order nodes and machine nodes. The policy value network module is used to generate multiple feasible schedules based on the action embedding using a policy gradient algorithm of maximum entropy reinforcement learning. The feasible schedules include at least the scheduling tasks of machine nodes executing work order nodes. The action sampling constraint module is used to sample or greedily select from multiple feasible schedules through a policy network and determine the specific schedule within seconds. The environment interaction and state transition module is used to adjust the heterogeneous graph according to the specific schedule, and to obtain updated state information when the specific schedule is detected to be executed. The reward signal generation module is used to evaluate the specific schedule being executed based on preset optimization goals and updated status information, and to generate reward signals. The training module is used to optimize and adjust the policy network in real time based on the reward signal and the maximum entropy reinforcement learning algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images