Using deep reinforcement learning for time constrained management at manufacturing systems

Abstract

A method for training an agent for a substrate manufacturing system is provided. The method includes: initializing an agent of a prediction subsystem of a substrate manufacturing system to select an action to be performed in a simulation environment associated with the substrate manufacturing system; and initiating a simulation of the selected action in the simulation environment. In response to suspending the simulation, the method further includes obtaining output data based on an environmental state associated with the simulation; and based on the output data, updating the agent to be configured to generate one or more scheduling decisions indicating a time at which processing of one or more substrates is initiated in the substrate manufacturing system.

Description

Technical Field

[0001] The present disclosure relates to methods and mechanisms for using deep reinforcement learning for time constraint management at manufacturing systems. Background Art

[0002] Before a substrate becomes a finished product (such as a wafer, an electronic device, etc.), the substrate can be processed according to a set of operations, each of which is performed at a tool in the manufacturing system. In some cases, one or more operations may be subject to time constraints. Time constraints refer to a specific amount of time to complete a subsequent operation after completing an operation. For example, a substrate can be processed according to a first operation and a second operation, in which a first material is deposited on the surface of the substrate, and in the second operation, a second material is deposited on the first material. The first operation and the second operation may be subject to time constraints, in which the second material is deposited on the first material within a specific amount of time, otherwise the first material may begin to degrade and the substrate cannot be used to produce a finished product (i.e., becomes unusable). The time constraint window refers to a specific amount of time to complete the operation that causes the time constraint (called the start operation), and the amount of time to complete the subsequent operation (called the completion operation) after the start operation is completed. In some cases, one or more operations can be performed between the start operation and the completion operation.

[0003] In most cases, when a substrate arrives at a tool, an operation cannot be started on the substrate because the tool may be processing other substrates. Therefore, an operator of a manufacturing system (e.g., an industrial engineer, a process engineer, a system engineer, etc.) schedules operations to run at specific times to meet time constraints associated with the operations. For example, the operator may postpone the operation on the substrate until each tool that is set to perform the operation associated with the time constraint has the capacity to perform the operation within the time constraint window.

[0004] In some cases, the completion operation of the first time constraint window can also be the start-up operation of the second time constraint window. In this case, the operator of the manufacturing system can schedule the start-up operation of the first time constraint window to start at a specific time to meet the first time constraint of the first time constraint window and the second time constraint of the second time constraint window. In other cases, an operation can be the completion operation of both the first time constraint window and the second time constraint window. In this case, the operator can schedule the start-up operation of the first time constraint window and the second time constraint of the second time constraint window to start at a specific time to meet the first time constraint of the first time constraint window and the second time constraint of the second time constraint window.

[0005] As manufacturing systems become more and more complex, more and more operations are subject to time constraints. In order to schedule a substrate to start at the start-up operation, an operator (e.g., using a computing system) must consider all time constraints that may be caused by the start-up operation. In order to take into account all time constraints that may be caused by the start-up operation, the operator must take into account the capacity of each tool that can perform the start-up operation, the completion operation, and each operation in between. In some cases, the time constraint window including the start-up operation may correspond to a considerable time (e.g., 6 hours, 8 hours, 12 hours, 24 hours, etc.). It may be difficult for the operator to take into account each time constraint and the capacity of each tool of the manufacturing system for a considerable period of time in the future. For some computing systems, this consideration can be classified as an NP-hard (non-deterministic polynomial-time hard) problem. Therefore, the operator may not be able to successfully schedule the substrate to start at each start-up operation of a set of operations so that each time constraint is met. Therefore, the substrate may violate the time constraints of a set of operations and become unavailable. Each substrate that becomes unavailable may reduce the overall system output and cause an increase in overall system delay. Summary of the invention

[0006] The following is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to identify the key or important elements of the disclosure, nor is it intended to delimit any scope of a particular embodiment of the disclosure or any scope of the claims. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to a more detailed description presented later.

[0007] In one aspect of the present disclosure, a method for training a software agent is provided. The method includes: initializing the software agent to select an action to be performed in a simulation environment associated with a manufacturing system; and starting a simulation of the selected action in the simulation environment. In response to pausing the simulation, the method further includes: obtaining output data based on an environment state associated with the simulation; and updating the software agent to be configured to generate one or more scheduling decisions based on the output data, the one or more scheduling decisions indicating a time to start processing of one or more substrates in the manufacturing system.

[0008] In another aspect of the present disclosure, a method for time constraint management at a manufacturing system is provided. The method includes: receiving a request to initiate a set of operations to be performed on a candidate set of substrates at a manufacturing system, wherein the set of operations includes one or more operations, each operation having one or more time constraints. The method further includes: obtaining current data related to a current state of the manufacturing system; and applying a software agent to the current data to determine a time to process the candidate set of substrates. The method further includes: initiating the set of operations on the candidate set of substrates at the determined time.

[0009] Another aspect of the present disclosure includes an electronic device manufacturing system including a memory device and a processing device operably coupled to the memory device to perform operations in accordance with any aspect or implementation described herein.

[0010] Another aspect of the present disclosure includes a non-transitory computer-readable storage medium including instructions that, when executed by a processing device operably coupled to a memory, perform operations according to any aspect or implementation described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present disclosure is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings.

[0012] Figure 1 is a block diagram illustrating an exemplary system architecture according to certain embodiments.

[0013] Figure 2 An example system for performing reinforcement learning to generate a software agent is described in accordance with certain embodiments.

[0014] Figure 3 is a flow chart of a method for training a software agent, according to certain embodiments.

[0015] Figure 4 is a top view of an example manufacturing system according to certain embodiments.

[0016] Figure 5 A set of operations subject to one or more time constraints in accordance with an embodiment of the present disclosure is described.

[0017] Figure 6 is a flow chart showing a method for initiating a set of operations based on scheduling decisions generated using a machine learning model, according to certain embodiments.

[0018] Figure 7 is a block diagram illustrating a computer system according to some implementations. DETAILED DESCRIPTION

[0019] Described herein are techniques for using reinforcement learning for time constraint management at a manufacturing system. In some processes, a series of operations may be performed at various stages of a manufacturing system. For example, a series of operations may be performed to deposit a coating (or multiple coatings) on the surface of a substrate and etch a three-dimensional pattern into the coating. In some cases, one or more of the series of operations may be subject to a time constraint. A time constraint may refer to a restriction or protocol in which, after an operation is performed at a manufacturing system, subsequent operations are to be completed within a specific amount of time. For example, a manufacturing system may be subject to a time constraint in which an etching process is to perform etching on a substrate within a specific number of hours (e.g., 12 hours) after a coating is deposited on the surface of a substrate. If the time constraint is not met (e.g., if the etching process is not performed within a specific number of hours), the substrate becomes defective and unusable.

[0020] Embodiments of the present disclosure relate to using deep reinforcement learning to manage time constraints at a substrate manufacturing system. A processing device may receive a request to initiate an operation to be run at a manufacturing system, wherein one or more operations are subject to time constraints. The processing device may determine when to release some substrates for processing based on the time constraints so that the substrates can be successfully processed at the manufacturing system within a specific time period. For example, the processing device may identify a set of candidate substrates at a substrate manufacturing system to be processed during the set of operations.

[0021] In order to identify a set of candidate substrates, the processing device can obtain data related to the current state of the manufacturing equipment. These data may include current state data, sensor data, context data, task data, etc. For example, the current data may be related to the following items: one or more operations performed on one or more substrates being processed, the number of substrates being processed at the manufacturing equipment at a specific moment, the number of substrates in the manufacturing equipment queue, the current service life, setting data, a set of operations including various processes performed at one or more manufacturing facilities in the production environment, sensor data, etc. The processing device can provide data related to the current state of the manufacturing equipment as input to the agent. The agent may include a software program that can perceive its environment, autonomously take actions to achieve one or more goals, and can use learning to improve its performance.

[0022] An agent (also referred to herein as software or intelligent agent) may be used to generate scheduling decisions. Scheduling decisions may determine what actions should be performed at a given time in a production environment. Examples of scheduling decisions may include, but are not limited to, where a substrate should be processed next in a production environment, which substrate should be selected for a piece of equipment that is idle in a production environment, and the like. Based on the scheduling decision data, a processing device may initiate the set of operations on a candidate set of substrates at a specific time.

[0023] In some embodiments, the scheduling decision may indicate at which time a group of candidate substrates are processed (e.g., when a substrate is scheduled to be started at a start operation). In other embodiments, the scheduling decision may involve decisions such as whether to start processing a batch with less than the allowed number of substrates, or to wait until additional substrates are available so that a full batch can be started before starting the batch. In other embodiments, the scheduling decision may involve determining to release substrates waiting at a logic gate step so that they can be scheduled for processing in a subsequent processing step. In some cases, in order to manage time constraints, the process flow will include a non-processing (logic) step (called a gate step) before the step that starts the time constraint. The lots (groups of substrates) wait at the gate step until the system determines that there is capacity for them to be fully processed. When there is capacity, they are released from the gate step and are available for processing at the first operation in the time constraint. A software agent can control this gate step.

[0024] In some embodiments, software agents can be trained using deep reinforcement learning. Deep reinforcement learning combines artificial neural networks with the architecture of reinforcement learning to help software agents learn how to achieve their goals (for example, deep reinforcement learning includes learning from existing knowledge and applying learning from existing knowledge to new data sets). In one example, during training, the software agent selects and simulates actions for one time step in the future (in a simulated environment). The software agent then receives new environmental states, as well as rewards. The sequence of state-action-reward is saved, and the reinforcement learning algorithm periodically uses this experience to update the weights of a neural network representing a policy. The policy is used to select the next action. The goal of the policy update is to maximize the cumulative reward over a time frame. Once the learning curve stabilizes and the policy stops improving, the policy is saved and can be used for current data related to manufacturing equipment.

[0025] Aspects and embodiments of the present disclosure address the shortcomings of the prior art by providing a technique for scheduling a substrate or a group of substrates to start at a startup operation. A processing device may use a trained software agent to determine a group of candidate substrates to be processed during a current or future time period (substrate group operation). By applying the software agent, the processing device may obtain a scheduling decision indicating when to schedule a group of substrates for processing. By determining when to schedule the group of substrates, the processing device may schedule the group of substrates to start at the group operation during the time period so that few or no substrates violate the time constraints of the group operation. Therefore, the number of substrates that violate the time constraints of the group operation is small or almost zero, resulting in a large number of substrates processed at the manufacturing system without defects or with few defects. Therefore, compared to traditional heuristic solutions that may reduce production, the trained software agent can reduce queue time violations while maintaining high production.

[0026] Figure 1 is a block diagram illustrating a production environment 100 according to aspects of the present disclosure. The production environment 100 may include multiple systems, such as and not limited to a production scheduler system 103, manufacturing equipment 112 (e.g., manufacturing tools, automation equipment, etc.), client devices 114, a prediction system 116 (e.g., for generating prediction data such as scheduling decisions, providing adaptation of models or agents, using knowledge bases, etc.), and one or more computer integrated manufacturing (CIM) systems 101. Examples of the production environment 100 may include and not be limited to manufacturing plants, fulfillment centers, etc. For the sake of simplicity, a manufacturing system is used as an example of the production environment 100 in any part of this description.

[0027] In some embodiments, the production environment 100 may be a semiconductor manufacturing environment. In such embodiments, the manufacturing equipment 112 may perform a plurality of different operations related to the manufacture of semiconductor substrates. For example, the manufacturing equipment 112 may perform cutting operations, cleaning operations, deposition operations, etching operations, testing operations, etc. Aspects of the present disclosure are descriptions of manufacturing semiconductor substrates in a semiconductor manufacturing environment. However, it should be noted that embodiments of the present disclosure may be applied to other production environments 100 configured to manufacture or otherwise process lots different from semiconductor substrates. A lot may refer to a group of substrates.

[0028] The manufacturing equipment 112 may include sensors 126 configured to capture data of substrates being processed in the manufacturing equipment 112. In some embodiments, the manufacturing equipment 112 and the sensors 126 may be part of a sensor system that includes a sensor server (e.g., a field service server (FSS) at a manufacturing facility) and a sensor identifier reader (e.g., a front opening pod (FOUP) radio frequency identification (RFID) reader of the sensor system). In some embodiments, the manufacturing equipment 112 may include or be operably coupled to a metrology equipment that includes a metrology server (e.g., a metrology database, a metrology folder, etc.) and a metrology identifier reader (e.g., a FOUP RFID reader of the metrology system).

[0029] The manufacturing equipment 112 can produce products, such as electronic devices, according to a recipe or by performing a trip over a period of time. The manufacturing equipment 112 may include a processing chamber. The manufacturing equipment 112 may perform a process on a substrate (e.g., a wafer, etc.) at the processing chamber. Examples of substrate processes include deposition processes for depositing one or more layers of a film on the surface of a substrate, etching processes for forming patterns on the surface of a substrate, and the like. The manufacturing equipment 122 may perform each process according to a process recipe. The process recipe defines a specific set of operations to be performed on a substrate during a process, and may include one or more settings associated with each operation. For example, a deposition process recipe may include a temperature setting for a processing chamber, a pressure setting for a processing chamber, a flow rate setting for a precursor of a material included in a film deposited on a substrate surface, and the like.

[0030] In some embodiments, the sensor 126 provides sensor data (e.g., sensor values, features, tracking data) associated with the manufacturing equipment 112 (e.g., associated with the production of a corresponding product (e.g., a wafer) by the manufacturing equipment 112). The manufacturing equipment 112 can produce a product according to a recipe or by performing a job over a period of time. The sensor data received over a period of time (e.g., corresponding to at least a portion of a recipe or operation) can be referred to as tracking data (e.g., historical tracking data, current tracking data, etc.) received from different sensors 126 over time. The sensor data may include values ​​of one or more of the following: temperature (e.g., heater temperature), spacing (SP), pressure, high frequency radio frequency (HFRF), voltage of an electrostatic chuck (ESC), current, material flow, power, voltage, etc. The sensor data may be associated with or indicate a manufacturing parameter, such as a hardware parameter, such as a setting or component (e.g., size, type, etc.) of the manufacturing equipment 124, or a process parameter of the manufacturing equipment 112. The sensor data may be provided when the manufacturing equipment 112 is performing a manufacturing process (e.g., equipment readings when processing a product). The sensor data for each substrate may be different.

[0031] The CIM 101, the production scheduler system 103, the manufacturing equipment 112, the client device 114, the forecasting system 116, and the data storages 140, 150 may be coupled to each other via a network 120. The network 120 may include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof. The CIM system 101, the production scheduler system 103, and the forecasting system 116 may be hosted separately or together in any combination by any type of machine, including a server computer, a gateway computer, a desktop computer, a laptop computer, a tablet computer, a notebook computer, a PDA (personal digital assistant), a mobile communication device, a cell phone, a smart phone, a handheld computer, or a similar computing device. In some embodiments, the forecasting system 116 is part of a server hosted on a machine.

[0032] The data storage 140, 150 may be a memory (e.g., random access memory), a drive (e.g., a hard drive, a flash drive), a database system, or another type of component or device capable of storing data. The data storage 140, 150 may include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers).

[0033] The data storage 140 may store data associated with processing a substrate at the manufacturing equipment 112. For example, the data storage 140 may store data collected by the sensor 126 at the manufacturing equipment 112 before, during, or after processing of a substrate (referred to as process data). The process data may refer to historical process data (e.g., process data generated for a previous substrate processed at the manufacturing system) and / or current process data (e.g., process data generated for a current substrate processed at the manufacturing system). The data storage may also store spectral data or non-spectral data associated with a portion of a substrate processed at the manufacturing equipment 112. The spectral data may include historical spectral data and / or current spectral data.

[0034] The data storage 140 may also store contextual data associated with one or more substrates processed at the manufacturing system. The contextual data may include a recipe name, a recipe step number, a preventive maintenance indicator, an operator, etc. The contextual data may refer to historical contextual data (e.g., contextual data associated with a previous process performed for a previous substrate) and / or current process data (e.g., contextual data associated with a current process or a future process to be performed for a previous substrate). The contextual data may include identification sensors associated with a particular subsystem of a processing chamber.

[0035] The data storage 140 may also store task data. Task data may include one or more groups of operations performed on a substrate during a deposition process, and may include one or more settings associated with each operation. For example, task data for a deposition process may include a temperature setting for a processing chamber, a pressure setting for a processing chamber, a flow rate setting for a precursor of a film material deposited on a substrate, and the like. In another example, task data may include controlling the pressure at a pressure point defined for a flow value. Task data may refer to historical task data (e.g., task data associated with a previous process performed for a previous substrate) and / or current task data (e.g., task data associated with a current process or a future process to be performed for a substrate).

[0036] In some embodiments, the data storage 140 may be configured to store data that is inaccessible to users of the manufacturing system. For example, a user of the manufacturing system (e.g., an operator) may not be able to access process data, spectral data, contextual data, etc. obtained for a substrate being processed at the manufacturing system. In some embodiments, all data stored at the data storage 140 may be inaccessible to users of the manufacturing system. In other or similar embodiments, a portion of the data stored at the data storage 140 may be inaccessible to the user, while another portion of the data stored at the data storage 140 may be accessible to the user. In some embodiments, one or more portions of the data stored at the data storage 140 may be encrypted using an encryption mechanism unknown to the user (e.g., the data is encrypted using a private encryption key). In other or similar embodiments, the data storage 140 may include multiple data storages, wherein the data that is inaccessible to the user is stored in one or more first data storages, and the data that is accessible to the user is stored in one or more second data storages.

[0037] The data store 150 includes scheduling rules 151, status data 153, and user data 155. The scheduling rules 151 can be logic that can be executed by the production scheduler system 103. In some embodiments, the scheduling rules 151 can be defined by a user (e.g., an industrial engineer, a process engineer, a system engineer, etc.). Examples of scheduling rules 151 can include, but are not limited to, selecting the highest priority substrate for the next step, selecting a substrate that uses the same settings as the settings targeted by the current configuration tool, packaging the item when the purchase order is completed, shipping the item when packaging is completed, and the like. Individual scheduling rules 151 can be associated with a large number of data processing used to implement the corresponding scheduling rule 151. Examples of data processing can include, but are not limited to, importing data, compressing data, indexing data, filtering data, performing mathematical functions on data, and the like.

[0038] The state data 153 may include the state of the manufacturing equipment 112 (e.g., operating temperature, operating pressure, the number of substrates being processed at the manufacturing equipment, the number of substrates in the manufacturing equipment queue at a given time, the current service life, setting data, a set of operations including various processes performed at one or more manufacturing facilities of the production environment, etc.). The state data 153 may be generated by the manufacturing equipment 112 during the operation of the production environment 100 and stored at the data storage 150. The state data 153 may include one or more of current state data, historical state data, and disturbed state data. The current state data may include data related to the current state of the manufacturing equipment 112 (e.g., the current operating temperature, the current operating pressure, the number of substrates currently being processed at the manufacturing equipment, etc.). The historical state data may include data related to the past state of the manufacturing equipment 112 (e.g., the past operating temperature at a specific time, the past operating pressure at a specific time, the past number of substrates being processed at the manufacturing equipment at a specific time, etc.). The disturbed state data may include modified state data. In particular, the disturbance state data may include current or historical state data having one or more parameters that are modified or distorted. The one or more parameters may be modified based on user input, a certain percentage, a certain value, randomly modified, etc. For example, the number of past substrates being processed at the manufacturing equipment at a particular moment of the disturbance state data may be reduced or increased by a predetermined value of two substrates. In another example, the number of past substrate sets being processed at the manufacturing equipment at a particular moment of the disturbance state data may be reduced or increased by a random set number between, for example, one and ten. In some embodiments, the state data 153 may include data stored in the data storage 140 or be generated by the data. For example, the state data 153 may include sensor data, context data, task data, etc. or be generated by sensor data, context data, task data, etc.

[0039] In some embodiments, state data may refer to data related to the environmental state of a simulation environment (e.g., environment 204). Environmental state data may include manufacturing equipment properties (e.g., step processing time, queue time constraints, etc.), manufacturing equipment observations (e.g., the number of substrates or batches processed per step, the number of batches processed per station, etc.), queue time observations (e.g., the number of batches successfully processed, the number of batches in violation, the number of batches in process, etc.), and capacity observations (e.g., an estimate of the time to complete all work in progress (WIP)). Environmental state features may be normalized to values ​​in [0,1] and concatenated into a single observation vector.

[0040] User data 155 may include data provided by a user of production environment 100 (eg, an operator, a process engineer, an industrial engineer, a system engineer, etc.) In some implementations, user data 155 may be provided via client device 114 .

[0041] The user device 114 may include a computing device such as a personal computer (PC), a laptop computer, a mobile phone, a smartphone, a tablet computer, a netbook computer, a network-connected television, etc. In some embodiments, the user device 114 may provide information to a user (e.g., an operator, an industrial engineer, a process engineer, a system engineer, etc.) of the production environment 100 via one or more graphical user interfaces (GUIs).

[0042] Examples of CIM systems 101 may include, but are not limited to, manufacturing execution systems (MES), enterprise resource planning (ERP), production planning and control (PPC), computer-aided systems (e.g., design, engineering, manufacturing, process planning, quality assurance), computer numerical control machines, direct numerical control machines, controllers, etc.

[0043] In some implementations, the prediction system 116 includes a prediction server 118 and a server machine 180. The prediction server 118 and the server machine 180 may each include one or more computing devices, such as a rack server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a graphics processing unit (GPU), an accelerator application specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc.

[0044] The prediction system 116 can train a software agent 190 (e.g., an intelligent agent). A software agent is a computer program that acts on behalf of a user or other program. In some embodiments, the software agent 190 can be trained using reinforcement learning, deep reinforcement learning, etc. Reinforcement learning is a class of algorithms that are suitable for sequential decision-making tasks. In particular, reinforcement learning is a process by which a software agent learns to make decisions through trial and error.

[0045] In some embodiments, training the software agent may include using deep reinforcement learning. Deep reinforcement learning combines artificial neural networks with the architecture of reinforcement learning to help software agents learn how to achieve their goals. In particular, deep reinforcement learning combines function approximation and goal optimization to map states and actions to the rewards they result in. Deep reinforcement learning includes learning from existing knowledge and applying learning from existing knowledge to new data sets, while reinforcement learning may include dynamically learning using trial and error to maximize results. In one embodiment, the software agent 190 may be trained using a proximal policy optimization (PPO) algorithm. The PPO algorithm is a deep RL algorithm that uses a policy gradient method to train a stochastic policy in an on-policy manner. The PPO algorithm also utilizes the actor-critic method. Figure 2 and 3 Details regarding training software agent 190 using deep reinforcement learning are described in .

[0046] Deep learning is a class of machine learning algorithms that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchy of layers, with different layers learning different levels of representation corresponding to different levels of abstraction. In deep learning, each layer learns to transform its input data into a slightly more abstract and comprehensive representation. Notably, the deep learning process can learn on its own which features to best place at which layer. The "depth" in "deep learning" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a considerable credit assignment path (CAP) depth. A CAP is a chain of transformations from an input to an output. A CAP describes the underlying causal connection between an input and an output. For a feedforward neural network, the depth of the CAP can be the depth of the network and can be the number of hidden layers plus 1. For a recurrent neural network, where a signal can propagate through a layer more than once, the CAP depth can be infinite.

[0047] Training of neural networks can be accomplished in a supervised learning fashion, which involves feeding a training dataset consisting of labeled inputs through the network, observing its output, defining an error (defined by measuring the difference between the output and the label value), and using techniques such as deep gradient descent and backpropagation to adjust the weights of all layers and nodes of the network to minimize the error. In many applications, repeating this process for many labeled inputs in the training dataset results in a network that can produce correct outputs when presented with inputs that differ from those present in the training dataset.

[0048] In some embodiments, the training of the neural network can be implemented using reinforcement learning. Reinforcement learning differs from supervised learning in that no labeled input / output pairs need to be presented, nor does it need to explicitly correct suboptimal actions. The focus of reinforcement learning can be to find a balance between exploring unknown areas and leveraging existing knowledge. Partially supervised reinforcement algorithms can combine the advantages of supervised algorithms and RL algorithms.

[0049] The server machine 180 may include a training engine 182. An engine may refer to hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, a processing device, etc.), software (e.g., instructions running on a processing device, a general purpose computer system, or a dedicated machine), firmware, microcode, or a combination of the above. The training engine 182 is capable of training one or more software agents 190. The software agents 190 may be generated by the training engine 182 using training data (also referred to herein as a training set), which may include a simulated environment, rewards, actions, states (e.g., observations), etc.

[0050] To implement training, the processing logic may input a training data set into one or more simulation environments. Before the first input is input into the simulation environment, the software agent may be initialized. The processing logic trains the software agent based on the actions provided to the simulation environment and the rewards and observations (based on the simulation state) obtained from the simulation environment. The processing logic may pause the simulation, and the software agent processes the obtained observations (e.g., state data) and reward data, and selects new actions to input into the simulation. The simulation then continues, which may be repeated until the simulation is complete. The software agent may be trained on multiple simulations. Once trained, the software agent may be applied to the current state data of the manufacturing equipment and generate outputs indicating one or more predictions or inferences. For example, the output predictions or inferences may include whether a candidate set of substrates can initiate time-sensitive constraints within a predetermined amount of time (e.g., 15 minutes in the future), when to release one or more substrates for processing, and so on.

[0051] After one or more rounds of training, processing logic can determine whether a stopping criterion has been met. The stopping criterion can be a target level of accuracy, a target number of processed images from the training data set, a target amount of change in a parameter relative to one or more previous data points, combinations of the above, and / or other criteria. In one embodiment, the stopping criterion is met when at least a minimum number of data points have been processed and at least a threshold accuracy has been achieved. The threshold accuracy can be, for example, 70%, 80%, or 90% accuracy. In one embodiment, the stopping criterion is met if the accuracy of the machine learning model has stopped improving. If the stopping criterion is not met, further training is performed. If the stopping criterion is met, training may be complete. Once the machine learning model is trained, a retained portion of the training data set can be used to test the model.

[0052] Once one or more trained software agents 190 are generated, they may be stored in the prediction server 118 as a prediction component 119 or a component of a prediction component 119 .

[0053] As described in detail below, the prediction server 118 includes a prediction component 119 that is capable of running a trained software agent 190 on current state data and providing prediction data indicating when one or more substrates are released for processing, the number of substrates that can be successfully processed at the manufacturing system according to a set of operations having one or more time constraints, etc. This will be explained in further detail below.

[0054] It should be noted that in some other implementations, the functionality of the server machine 180 and the prediction server 118 may be provided by a smaller number of machines. For example, in some implementations, the server machine 180 and the prediction server 118 may be integrated into a single machine.

[0055] Generally speaking, functions described as being performed by the server machine 180 and / or prediction server 118 in one embodiment may also be performed on the client device 114. Furthermore, functionality attributed to a particular component may also be performed by a different component or components operating together.

[0056] In an embodiment, a "user" may be represented as a single individual. However, other embodiments of the present disclosure also include a "user" that is an entity controlled by multiple users and / or automated sources. For example, a group of individual users united as a group of administrators may be considered a "user".

[0057] The production scheduler system 103 can make scheduling decisions for the production environment 100. Scheduling decisions determine what actions should be performed at a given time in the production environment 100. Scheduling typically involves decisions such as whether to start processing a batch, whether to start processing a batch with fewer substrates than allowed, or to wait until additional substrates are available so that a full batch can be started, etc. Examples of scheduling decisions can include, but are not limited to, where a substrate should be processed next in the production environment, which substrate should be picked for a piece of equipment that is idle in the production environment, etc. In some embodiments, the production scheduler system 103 can use forecasting data generated by the forecasting component 119 to make scheduling decisions. In some embodiments, the production scheduler system 103 can use one or more scheduling rules 151 stored in the data storage 150 to make scheduling decisions.

[0058] In some cases, the manufacturing process may include hundreds of operations performed by manufacturing equipment 112 (e.g., tools or automated equipment) within the production environment 100. In many cases, one or more operations may be subject to time constraints. As previously discussed, a time constraint refers to a specific amount of time to complete a subsequent operation after completing an operation. For example, after a first material is deposited on the surface of a substrate, a second material is to be deposited on the first material within a specific amount of time after the deposition of the first material. If the second coating is not deposited on the first material within a specific amount of time, the first material may begin to degrade, making the substrate unusable. The time constraint window refers to the amount of time to complete the first operation (referred to as the start operation) and the specific amount of time to complete the second operation (referred to as the completion operation). In some embodiments, one or more operations performed between the start operation and the completion operation are also associated with the time constraint window. Based on the previous example, the time constraint window may refer to the first amount of time to deposit the first material on the surface of the substrate and the specific amount of time to deposit the second material on the first material. Multiple operations may be subject to one or more time constraints. In some embodiments, the completion operation of the first time constraint window may also be the start operation of the second time constraint window.

[0059] Figure 2An example system 200 for performing reinforcement learning to generate a software agent is illustrated in accordance with certain embodiments of the present disclosure. The example system 200 includes a software agent 202 and a simulation environment 204 (e.g., a simulator). The agent 202 takes actions that affect the environment 204 and change its state (e.g., the environment state). The environment state is a representation of the current environment in which the agent is located. This state can be observed by the agent 202, and it includes all relevant information about the environment that the agent 202 needs to know in order to make a decision (e.g., perform an action). After each action, the agent 202 transitions to the next state of the environment and receives a reward.

[0060] The agent 202 may use one or more machine learning models 240. The machine learning model 240 may be, for example, a deep neural network (e.g., a convolutional neural network, a transformer, a graph neural network, etc.) or a decision tree. The machine learning model 240 may represent a strategy (e.g., a solution strategy). The strategy may be an action strategy that is expected to obtain the highest long-term reward.

[0061] The agent 202 may be rewarded for taking control of an environment state that results in a successful one. Rewards may be immediate, such as one point for each step taken in the right direction, or they may be delayed, such as one point at the end of an episode if a goal is reached. An episode may refer to a sequence of environment states, actions, and rewards, ending with a terminal environment state. In an illustrative example, each episode (or experiment) may include 100 timesteps, each of which may be 100 minutes. At each timestep, the agent 202 may take a single action. After the action, the agent 202 receives an observation (e.g., environment state data) that reflects the state of the environment 204 at the end of the timestep. An episode ends when 100 timesteps have passed, or, for example, when a predetermined number of batches (e.g., 10 batches) complete the route, whichever occurs first.

[0062] In some embodiments, the example system 200 uses the Markov Decision Process (MDP) formalism, where the agent 202 attempts to optimize a function in its environment 204. The MDP can be composed of an environment state space S (denoted as s∈S), an action space A (a∈A), a transition function T: S×A→S, and a reward function Description. In an MDP, an instance evolves over discrete time steps t=0, 1, 2, ..., n, where the agent 202 observes the environment state s t (206) and use the strategy π(a t |s t ) Use action at (210) Response. The environment 204 provides the agent 202 with the next environment state s t+1 ~T(s t , a t )212 and reward r t =R(s t , a t ) 214. The task of the agent 202 is to maximize the reward (accumulated future reward) by learning the best policy π*.

[0063] In some embodiments, queue time management can be modeled as a tuple M = (S, A, P, R, ρ 0 , T) of discontinuous time, finite-horizon MDP, where S is the set of environment states, A is the set of actions, P: S×A×S→R+ is the transition probability distribution, R: S×A→R is the reward function, ρ 0 :S→[0,1] is the initial environment state distribution, T is the time horizon. The solution strategy can be a probability distribution π that maps the environment state to the action: S×A→[0,1]. In order to find the solution strategy, the agent 202 can be trained to learn a strategy that maximizes the expected reward. strategy, where τ: = (s 0 , a 0 ,s 1 , a 1 ...) represents the trajectory, s 0 ~ρ 0 , a t ~π(s t ), s t+1 ~P(s t , a t ).

[0064] During training, the agent 202 takes an action. The environment 204 applies the action and simulates one time step into the future. The agent 202 then receives new environment state data and a new reward. The sequence of state-action-reward is stored, and the reinforcement learning algorithm periodically uses this experience to update the weights of the neural network (e.g., machine learning model 240) that represents the policy. The policy is used to select the next action. The goal of the policy update is to maximize the cumulative reward over a time horizon. Once the learning curve stabilizes and the policy stops improving, the processing logic (e.g., training engine 182) can store the policy and use it to test the performance of the software agent 202 on one or more environments.

[0065] Environmental state data (e.g., data related to the state of the environment 204) may include manufacturing equipment properties (e.g., step processing time, queue time constraints, etc.), manufacturing equipment observations (e.g., number of substrates or batches processed per step, number of batches processed per station, etc.), queue time observations (e.g., number of batches successfully processed, number of batches in violation, number of batches in process, etc.), capacity observations (e.g., estimates of time to complete all work in process (WIP)), number of batches or substrates waiting to be processed at various steps and / or waiting to start various time constraints, etc. The state features may be normalized to values ​​in [0,1] and concatenated into a single observation vector.

[0066] At each time step, the agent 202 can decide to release or not release a batch. The agent 202 can release batches of one of N part types (or batches waiting for a processing step or a gate step). Therefore, the agent 202 can choose a discrete action between 0 and N. Selecting action 0 will not release any batches, and action a i Will release type Part i The agent 202 may also select an action involving releasing multiple batches, which may be of different part types or steps.

[0067] The compensation structure can be configured so that it encourages the agent 202 to minimize the number of queue time violations while optimizing the makespan (e.g., the time difference between the start and end of a series of jobs or tasks) and the number of successful batches. The compensation structure can also be configured so that it encourages the agent 202 to maximize the output of the manufacturing equipment.

[0068] Figure 3 is a flow chart of a method 300 for training a software agent according to aspects of the present disclosure. The method 300 is performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a general-purpose computer system or a dedicated machine), firmware, or some combination thereof. In one embodiment, the method 300 may be performed by a computer system (e.g., Figure 1 In other or similar embodiments, one or more operations of method 300 may be performed by one or more other machines not depicted in the drawings. In some aspects, one or more operations of method 300 may be performed by server machine 180 and / or prediction server 118.

[0069] For ease of explanation, these methods are depicted and described as a series of actions. However, actions according to the present disclosure may occur in various orders and / or in parallel, and occur together with other actions not proposed and described herein. In addition, the method according to the disclosed subject matter can be implemented without performing all the actions shown. In addition, those skilled in the art will appreciate and understand that these methods may alternatively be represented as a series of interrelated states via state diagrams or events. In addition, it should be understood that the methods disclosed in this specification can be stored on a manufactured article to facilitate transporting and transferring such methods to a computing device. The term manufactured article used herein is intended to include a computer program accessible from any computer-readable device or storage medium.

[0070] Processing logic initializes a software agent in operation 310. In some implementations, the software agent has access to environmental state data and or state data (eg, data associated with operations related to fabrication of semiconductor substrates, such as historical state data, current state data, perturbation state data, etc.).

[0071] In operation 312, processing logic performs one or more simulations. The one or more simulations may be performed in a simulation environment (e.g., environment 204). In some embodiments, the simulation may include simulating an action (e.g., a time step in the future). In some embodiments, processing logic may determine a specific time period for the operational training set to be run at the manufacturing system. In some embodiments, the operational training set may be Figure 5 According to the previously described embodiment, a specific time period may be a simulation condition.

[0072] In some embodiments, in response to the software agent selecting action data, a simulation may be performed. The action data may include a set of possible actions, movements, or operations that the software agent may perform. In some embodiments, the actions may include not releasing a batch, releasing a specific batch, releasing a batch for a specific processing chamber, releasing a batch during a certain time period, and the like. In some embodiments, the actions may include determining a training set of substrates to be processed during the operational training set. The candidate substrate training set and the operational training set are determined using state data, operator input, a predetermined set of rules (e.g., a predetermined set of one or more sets of substrates, a predetermined set of one or more sets of operations, etc.), random input, or any combination of the foregoing.

[0073] In operation 314, processing logic pauses the simulation to obtain output data. In some implementations, the output data may include new environment state data based on the current environment state and reward data.

[0074] In operation 316, processing logic updates the software agent based on the output data (eg, new environment state data and new reward data). The new reward data may include feedback data by which the success or failure of an action in a given state is measured.

[0075] In operation 318 , processing logic generates new action (eg, action data) data by the software agent based on the new state data.

[0076] In operation 320, processing logic continues simulation using the new action data. For example, processing logic may simulate new actions in the environment.

[0077] The processing logic may perform operations 312 to 316 until the simulation or a set of simulations is complete. The processing logic may perform operation 300 until the training of the software agent is complete. In some embodiments, the output data indicates the number of candidate substrates that were successfully processed during each operation in the set of operations of the simulation to reach the end of the time period.

[0078] It should be noted that in some embodiments, the adequacy of training may be determined based solely on the amount of training data or updates to the software agent, while in some other embodiments, the adequacy of training may be determined based on one or more other criteria (e.g., a measure of the diversity of training examples, etc.).

[0079] After operation 318, the software agent can be used to generate prediction data (e.g., scheduling decisions) based on the current state data. In some embodiments, the prediction data may include one or more scheduling decisions. For example, the machine learning model may receive the current state data as input and output a scheduling decision. As discussed above, the scheduling decision determines what action should be performed at a given time in the production environment 100. Scheduling may involve some decisions, such as whether to start processing a batch with less than the allowed number of substrates, or wait until additional substrates are available so that a full batch can be started before starting a batch. Examples of scheduling decisions may include, but are not limited to, where a substrate should be processed next in the production environment, which substrate should be selected for an idle piece of equipment in the production environment, and the like.

[0080] Figure 4 4 is a top view schematic diagram of an example manufacturing system 400 in accordance with aspects of the present disclosure. Manufacturing system 400 can perform one or more processes on substrate 402. Substrate 402 can be any suitable rigid, fixed-size, planar article, such as a disk or wafer containing silicon, a patterned wafer, a glass plate, or the like, suitable for manufacturing electronic devices or circuit components on substrate 402.

[0081] The manufacturing system 400 may include a processing tool 404 and a factory interface 406 coupled to the processing tool 404. The processing tool 404 may include a housing 408 having a transfer chamber 410 therein. The transfer chamber 410 may include one or more processing chambers (also referred to as process chambers) 414, 416, 418, which are arranged around and coupled to the transfer chamber. The processing chambers 414, 416, 418 may be coupled to the transfer chamber 410 through corresponding ports (e.g., slit valves, etc.). The transfer chamber 410 may also include a transfer chamber robot 412, which is configured to transfer the substrate 402 between the processing chambers 414, 416, 418, the load lock 420, etc. The transfer chamber robot 412 may include one or more arms, wherein each arm includes one or more end effectors at the end of each arm. The end effectors may be configured to carry specific objects, such as wafers, sensor disks, sensor tools, etc.

[0082] The processing chambers 414, 416, 418 can be adapted to implement any number of processes on the substrate 402. The same or different substrate processes can be performed in each processing chamber 414, 416, 418. The substrate process can include atomic layer deposition (ALD), physical vapor deposition (PVD), chemical vapor deposition (CVD), etching, annealing, curing, pre-cleaning, metal or metal oxide removal, or similar processes. Other processes can be implemented on the substrate therein. The processing chambers 414, 416, 418 can each include one or more sensors that are configured to capture data of the substrate 402 before, after, or during the substrate process. For example, the one or more sensors can be configured to capture spectral data and / or non-spectral data of a portion of the substrate 402 during the substrate process. In other or similar embodiments, the one or more sensors can be configured to capture data associated with the environment within the processing chambers 414, 416, 418 before, after, or during the substrate process. For example, the one or more sensors may be configured to capture data associated with the temperature, pressure, gas concentration, etc. of the environment within the processing chambers 414, 416, 418 during substrate processing.

[0083] A load lock 420 may also be coupled to the housing 408 and the transfer chamber 410. The load lock 420 may be configured to interface and couple with the transfer chamber 410 on one side and to interface and couple with the factory interface 406. In some embodiments, the load lock 420 may have an environmentally controlled atmosphere that may change from a vacuum environment (where substrates may be transferred to or from the transfer chamber 410) to an atmospheric pressure or near-atmospheric pressure inert gas environment (where substrates may be transferred to and from the factory interface 406). The factory interface 406 may be any suitable housing, such as an equipment front end module (EFEM). The factory interface 406 may be configured to receive substrates 402 from substrate carriers 422 (e.g., front opening pods (FOUPs)) docked at various load ports 424 of the factory interface 406. A factory interface robot 426 (shown in phantom) may be configured to transfer substrates 402 between carriers (also referred to as containers) 422 and the load lock 420. The carrier 422 may be a substrate storage carrier or a replacement parts storage carrier.

[0084] The manufacturing system 400 may also be connected to a client device (not shown) configured to provide information to a user (e.g., an operator) regarding the manufacturing system 400. In some embodiments, the client device may provide information to a user of the manufacturing system 400 via one or more graphical user interfaces (GUIs). For example, the client device may provide information via the GUI regarding a target thickness profile of a film to be deposited on a surface of the substrate 402 during a deposition process performed at the processing chambers 414, 416, 418. According to embodiments described herein, the client device may also provide information regarding modifications to a process recipe according to a corresponding set of deposition settings predicted to correspond to the target profile.

[0085] The manufacturing system 400 may also include a system controller 428. The system controller 428 may be and / or may include computing devices such as a personal computer, a server computer, a programmable logic controller (PLC), a microcontroller, etc. The system controller 428 may include one or more processing devices, which may be general processing devices such as a microprocessor, a central processing unit, or the like. In more detail, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. The processing device may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The system controller 428 may include a data storage device (e.g., one or more disk drives and / or solid-state hard disks), a main memory, a static memory, a network interface, and / or other components. The system controller 428 may execute instructions to execute any one or more methodologies and / or embodiments described herein. In some embodiments, the system controller 428 may execute instructions to perform one or more operations at the manufacturing system 400 according to the process recipe. The instructions may be stored on a computer-readable storage medium, which may include main memory, static memory, secondary storage, and / or a processing device (during execution of the instructions).

[0086] The system controller 428 can receive data from sensors included on or within various parts of the manufacturing system 400 (e.g., processing chambers 414, 416, 418, transfer chamber 410, load lock 420, etc.). In some embodiments, the data received by the system controller 428 can include spectral data and non-spectral data of a portion of the substrate 402. In other or similar embodiments, the data received by the system controller 428 can include data associated with processing the substrate 402 at the processing chambers 414, 416, 418, as previously described. For the purposes of this description, the system controller 428 is described as receiving data from sensors included within the processing chambers 414, 416, 418. However, the system controller 428 can also receive data from any part of the manufacturing system 400, and the data received from the part can be used in accordance with the embodiments described herein. In an illustrative example, the system controller 428 can receive data from one or more sensors of the processing chambers 414, 416, 418 before, after, or during a substrate process at the processing chambers 414, 416, 418. Data received from sensors in various parts of the manufacturing system 400 may be stored in a data storage 450. The data storage 450 may be included as a component within the system controller 428, or may be a separate component from the system controller 428. In some embodiments, the data storage 450 may be a Figure 1 The data storage devices 140, 150 are described.

[0087] Figure 5 A set of operations 500 subject to one or more time constraints is described in accordance with an embodiment of the present disclosure. Each operation 510 in the training set of operations may correspond to a separate process performed at one or more manufacturing facilities of a production environment, such as manufacturing equipment 112 (e.g., tools or automation equipment) of the production environment 100. In some embodiments, each operation in the set of operations 500 may be a continuous operation (e.g., each operation 510 is performed in a specific order). In some embodiments, each operation 510 may correspond to a separate process performed at a front-end manufacturing facility, including but not limited to lithography, deposition, etching, cleaning, ion implantation, chemical and mechanical polishing, etc. In other or similar embodiments, each operation may correspond to a separate process performed at a back-end manufacturing facility, including but not limited to dicing a finished wafer into separate semiconductor die, testing, assembly, packaging, etc.

[0088] As previously described, one or more operations 510 may be subject to a time constraint. For example, operation 2 may be a first deposition operation for depositing a first material on a substrate surface, and operation 3 may be a second deposition operation for depositing a second material on the first material. Operations 2 and 3 may be subject to a first time constraint, wherein the second material is to be deposited on the first material within a specific amount of time (e.g., 6 hours) after the first material is deposited on the substrate surface. The amount of time for the manufacturing equipment 112 to perform operations 2 and 3 may correspond to a time constraint window 512. The time constraint window 512 may include a first amount of time to complete the start operation (i.e., start operation 510 of the time constraint window 512), and a specific amount of time within which the manufacturing equipment 112 is to complete the completion operation (i.e., complete operation 510 of the time constraint window 512). Based on the previous example, operation 2 is to be initiated on a substrate at the manufacturing equipment 112, so that operations 2 and 3 will be completed on the substrate within the first time constraint window 512A.

[0089] In some embodiments, the completion operation of a time constraint window 512 may be the start operation of another time constraint window 512. For example, operation 3 may be a second deposition operation and operation 6 may be an etching operation. Operations 3, 4, 5, and 6 may be subject to a time constraint, wherein the second material is to be etched in operation 6 within a specific amount of time (e.g., 12 hours) after the second material is deposited in operation 3. The second time constraint window 512B may include the amount of time to deposit the second material in operation 3, and the specific amount of time to complete operation 6. Operation 3 is to be initiated at the manufacturing equipment 112 such that operations 3, 4, 5, and 6 will be completed within the second time constraint window 512B. Based on the previous example, operation 3 may be subject to the time constraint of operation 2. Therefore, operation 2 is to be initiated on the substrate such that operations 2 and 3 will be completed on the substrate within the first time constraint window 512A, and operations 3, 4, 5, and 6 will be completed within the second time constraint window 512B. The first time constraint window 512A and the second time constraint window 512B are collectively referred to as cascaded time constraint windows.

[0090] In some embodiments, operation 510 may be subject to more than one time constraint. For example, operations 6, 7, 8, 9, and 10 may be subject to a first time constraint, wherein operation 10 is to be completed within a specific amount of time after operation 6 is completed. The third time constraint window 512C may include the amount of time to perform operation 6 and the specific amount of time to complete operation 10. Operations 9 and 10 may also be subject to a second time constraint, wherein operation 10 is to be completed within a specific amount of time after operation 9 is completed. The fourth time constraint window 512D may include the amount of time to complete operation 9 and the specific amount of time to complete operation 10. Therefore, operation 6 is to be started so that operations 6, 7, 8, 9, and 10 will be completed within the third time constraint window 512D, and operations 9 and 10 will be completed within the fourth time constraint window. The third time constraint window 512C and the fourth time constraint window 512 are collectively referred to as a nested time constraint window.

[0091] Figure 6 is a flow chart of a method 600 for initiating a set of operations based on a scheduling decision generated using a software agent in accordance with aspects of the present disclosure. The method 600 is performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a general-purpose computer system or a dedicated machine), firmware, or some combination thereof. In one embodiment, the method 600 may be performed by a computer system (e.g., Figure 1 In other or similar embodiments, one or more operations of method 600 may be performed by one or more other machines not depicted in the figure. In some aspects, one or more operations of method 600 may be performed by server machine 180, forecast server 118, CIM system 101, and / or production scheduler system 103.

[0092] In operation 610, processing logic receives a request to initiate a set of operations to be performed at a manufacturing system. In some embodiments, the manufacturing system may be Figure 1 In some embodiments, the request may be a request to start the set of operations to be run at the manufacturing system at a specific time. For example, the request may be a request to start the set of operations at 8 pm. In some embodiments, the request may be a request to start the set of operations on a candidate set of substrates. In some embodiments, the request may be a request for a scheduling decision related to the candidate set of substrates. For example, the request may request the next available time to start the set of operations on the candidate set of substrates without time constraint issues.

[0093] In operation 612, the processing logic obtains current data related to the current state of the manufacturing equipment. In some embodiments, the current data may include current state data, sensor data, context data, task data, etc. In some embodiments, the current data may include the number of substrates being processed at the manufacturing equipment at a particular moment, the number of substrates in the manufacturing equipment queue, the current service life, setting data, a set of operations including various processes performed at one or more manufacturing facilities in the production environment, and the like. In some embodiments, the current data may be related to one or more operations performed on one or more substrates being processed. For example, the operation may include a deposition process performed in a processing chamber to deposit one or more layers of a film on the surface of a substrate, an etching process performed on the one or more layers of the film on the surface of the substrate, and the like. The operation may be performed according to a recipe. The sensor data may include values ​​of one or more of the following: temperature (e.g., heater temperature), interval, pressure, high frequency radio frequency, voltage of an electrostatic chuck, current, material flow, power, voltage, and the like. The sensor data may be associated with or indicate a manufacturing parameter, such as a hardware parameter, such as a setting or component (e.g., size, type, etc.) of the manufacturing equipment 112, or a process parameter of the manufacturing equipment 112.

[0094] In operation 614, processing logic applies a software agent (eg, agent 190) to the obtained current data. The software agent may be used to generate prediction data including one or more scheduling decisions.

[0095] In operation 616, processing logic initiates a set of operations at the manufacturing system to process the candidate set of substrates within a specified time period.

[0096] In some embodiments, the forecast data includes one or more scheduling decisions. A scheduling decision determines what action should be performed at a given time in the production environment 100. In some embodiments, a scheduling decision may include a candidate set of substrates and a specified time period. A software agent may generate the forecast data.

[0097] Figure 7700 is a block diagram illustrating a computer system 700 according to certain embodiments. In some embodiments, the computer system 700 can be connected to other computer systems (e.g., via a network connection, such as a local area network (LAN), an intranet, an extranet, or the Internet). The computer system 700 can operate as a server or client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 700 can be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web appliance, a server, a network router, a switch or a bridge, or any device capable of executing a set of instructions (in sequence or otherwise) that specify actions to be taken by the device. Further, the term "computer" shall include any collection of computers that execute a set (or multiple sets) of instructions individually or jointly to perform any one or more of the methods described herein.

[0098] In another aspect, the computer system 700 may include a processing device 702, a volatile memory 704 (e.g., a random access memory (RAM)), a non-volatile memory 706 (e.g., a read-only memory (ROM) or an electrically erasable programmable ROM (EEPROM)), and a data storage device 716, which may communicate with each other via a bus 708.

[0099] The processing device 702 may be provided by one or more processors such as a general purpose processor (e.g., a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor that implements other types of instruction sets, or a microprocessor that implements a combination of multiple types of instruction sets) or a special processor (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0100] The computer system 700 may further include a network interface device 722 (e.g., coupled to the network 774). The computer system 700 may also include a video display unit 710 (e.g., LCD), an alphanumeric input device 712 (e.g., keyboard), a cursor control device 714 (e.g., mouse), and a signal generating device 720.

[0101] In some embodiments, the data storage device 716 may include a non-transitory computer-readable storage medium 724 on which instructions 726 may be stored that encode any one or more of the methods or functions described herein, including Figure 1The components (eg, prediction component 119, time constraint simulation module 107, etc.) are encoded with instructions for implementing the methods described herein.

[0102] The instructions 726 may also reside, completely or partially, within the volatile memory 704 and / or the processing device 702 during execution of the instructions 726 by the computer system 700, and thus the volatile memory 704 and the processing device 702 may also constitute machine-readable storage media.

[0103] Although the computer-readable storage medium 724 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" shall also include a single medium or multiple media (such as a centralized or distributed database and / or associated caches and servers) storing the one or more sets of executable instructions. The term "computer-readable storage medium" shall also include any tangible medium capable of storing or encoding a set of instructions for execution by a computer, causing the computer to perform any one or more of the methods described herein. The term "computer-readable storage medium" shall include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0104] The methods, components and features described herein may be implemented by discrete hardware components, or may be integrated into the functionality of other hardware components such as ASICS, FPGAs, DSPs or similar devices. In addition, the methods, components and features may also be implemented by firmware modules or functional circuit systems within hardware devices. Further, the methods, components and features may be implemented in any combination of hardware devices and computer program components or in a computer program.

[0105] Unless otherwise specifically stated, terms such as "receiving", "executing", "providing", "obtaining", "causing", "accessing", "determining", "adding", "using", "training", etc. refer to actions and processes performed or implemented by a computer system that manipulate and transform data represented as physical (electronic) quantities within computer system buffers and memories into other data similarly represented as physical quantities within computer system memories or buffers or other such information storage, transmission or display devices. Also, the terms "first", "second", "third", "fourth", etc. used herein are used as labels to distinguish different elements and may not have a sequential meaning based on their numerical labels.

[0106] The examples described herein also relate to a device for performing the methods described herein. This device may be specially constructed to perform the methods described herein, or it may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0107] The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove appropriate to construct more specialized apparatus to perform the methods described herein and / or each of the individual functions, routines, subroutines, or operations of the methods. Examples of structures for various such systems are set forth in the description above.

[0108] The above description is intended to be illustrative rather than restrictive. Although the present disclosure has been described with reference to specific illustrative examples and embodiments, it will be appreciated that the present disclosure is not limited to the examples and embodiments described. The scope of the present disclosure will be determined with reference to the following claims and the full range of equivalents to which these claims are assigned.

Claims

1. A method comprising: Initializing, by a processor, an agent of a prediction subsystem of a substrate manufacturing system to select an action to be performed in a simulation environment associated with the substrate manufacturing system; Initiating simulation of the selected action in the simulation environment; In response to pausing the simulation, obtaining output data based on an environment state associated with the simulation; as well as Based on the output data, the agent is updated to be configured to generate one or more scheduling decisions indicating a time to initiate processing of one or more substrates in the substrate manufacturing system.

2. The method of claim 1, further comprising: receiving a request to initiate a set of operations to be performed at the substrate manufacturing system on a candidate set of substrates, wherein the set of operations includes one or more operations, each operation having one or more time constraints; obtaining current data related to a current state of the substrate manufacturing system; providing the current data as an input to the agent to obtain one or more outputs indicating a time to process a candidate of the set of substrates; as well as The set of operations is initiated on the candidate set of substrates at the determined time.

3. The method of claim 1, further comprising: receiving a request to initiate a set of operations to be performed at the substrate manufacturing system on a candidate set of substrates, wherein the set of operations includes one or more operations, each operation having one or more time constraints; obtaining current data related to a current state of the substrate manufacturing system; providing the current data as input to the agent to obtain one or more outputs indicating a subset of substrates to be processed from a candidate set of substrates; as well as The set of operations is initiated on the subset of substrates.

4. The method of claim 1, wherein the agent comprises a deep reinforcement learning model.

5. The method of claim 1, further comprising: selecting a new action based on the output data; as well as The simulation of the new action is initiated in the simulation environment.

6. The method of claim 1, wherein the output data comprises environmental status data and reward data, wherein the environmental status data comprises at least one of manufacturing equipment properties, manufacturing equipment observations, queue time observations, or capacity observations.

7. The method of claim 1, wherein the action comprises a decision to at least one of: initiate processing of one or more substrates, not initiate processing of the one or more substrates, or initiate processing of a subset of the one or more substrates.

8. An electronic equipment manufacturing system, comprising: Memory devices; as well as A processing device operatively coupled to the memory device to perform operations comprising the steps of: Initializing an agent of a prediction subsystem of the manufacturing system to select actions to be performed in a simulation environment associated with the manufacturing system; Initiating simulation of the selected action in the simulation environment; In response to pausing the simulation, obtaining output data based on an environment state associated with the simulation; as well as Based on the output data, the agent is updated to be configured to generate one or more scheduling decisions indicating a time to initiate processing of one or more substrates in the manufacturing system.

9. The electronic device manufacturing system of claim 8, wherein the operations further comprise: receiving a request to initiate a set of operations to be performed at the manufacturing system on a candidate set of substrates, wherein the set of operations includes one or more operations, each operation having one or more time constraints; obtaining current data related to a current state of the manufacturing system; providing the current data as an input to the agent to obtain one or more outputs indicating a time to process a candidate of the set of substrates; as well as The set of operations is initiated on the candidate set of substrates at the determined time.

10. The electronic device manufacturing system of claim 8, wherein the operation further comprises: receiving a request to initiate a set of operations to be performed at the manufacturing system on a candidate set of substrates, wherein the set of operations includes one or more operations, each operation having one or more time constraints; obtaining current data related to a current state of the manufacturing system; providing the current data as input to the agent to obtain one or more outputs indicating a subset of substrates to be processed from a candidate set of substrates; as well as The set of operations is initiated on the subset of substrates.

11. The electronic device manufacturing system of claim 8, wherein the agent comprises a deep reinforcement learning model.

12. The electronic device manufacturing system of claim 8, wherein the operations further comprise: selecting a new action based on the output data; as well as The simulation of the new action is initiated in the simulation environment.

13. The electronic device manufacturing system of claim 8, wherein the output data comprises environmental status data and reward data, wherein the environmental status data comprises at least one of manufacturing equipment properties, manufacturing equipment observations, queue time observations, or capacity observations.

14. The electronic device manufacturing system of claim 8, wherein the action comprises a decision to at least one of: initiate processing of one or more substrates, not initiate processing of the one or more substrates, or initiate processing of a subset of the one or more substrates.

15. A method comprising: receiving a request to initiate a set of operations to be performed at a substrate manufacturing system on a candidate set of substrates, wherein the set of operations includes one or more operations, each operation having one or more time constraints; obtaining current data related to a current state of the substrate manufacturing system; providing the current data as an input to the agent to obtain one or more outputs indicating a time to process a candidate of the set of substrates; as well as The set of operations is initiated on at least one of the candidate set of substrates at the determined time, or the set of operations is initiated on the subset of substrates.

16. The method of claim 15, wherein the step of training the agent comprises: Initializing the agent to select actions to be performed in a simulation environment associated with the substrate manufacturing system; Initiating simulation of the selected action in the simulation environment; In response to pausing the simulation, obtaining output data based on an environment state associated with the simulation; as well as Based on the output data, the agent is updated to be configured to generate one or more scheduling decisions indicating a time to initiate processing of one or more substrates in the substrate manufacturing system.

17. The method of claim 15, wherein the agent comprises a deep reinforcement learning model.

18. The method of claim 15, wherein the output data includes environmental status data and reward data.

19. The method of claim 18, wherein the environmental status data comprises at least one of manufacturing equipment properties, manufacturing equipment observations, queue time observations, or capacity observations.

20. The method of claim 15, wherein the action comprises a decision to at least one of: initiate processing of one or more substrates, not initiate processing of the one or more substrates, or initiate processing of a subset of the one or more substrates.

Images