Using Deep Reinforcement Learning for Time Constraint Management in Manufacturing Systems

Abstract

A method is provided for training an agent for a substrate manufacturing system. The method includes initializing an agent of a prediction subsystem of the substrate manufacturing system to select an action to perform in a simulated environment associated with the substrate manufacturing system and initiating a simulation of the selected action in the simulated environment. In response to pausing the simulation, the method further includes obtaining output data based on an environmental state associated with the simulation and updating the agent to be configured to generate one or more dispatching decisions indicative of a time to begin processing one or more substrates in the substrate manufacturing system based on the output data.

Description

[Technical field]

[0001] The present disclosure relates to methods and mechanisms for using deep reinforcement learning for time constraint management in manufacturing systems. [Background technology]

[0002] Before a substrate becomes a finished product (e.g., a wafer, electronic device, etc.), it may be processed according to a series of operations, each performed by a tool of a manufacturing system. In some cases, one or more operations may be subject to a time constraint. A time constraint refers to a specific amount of time that a subsequent operation must be completed after one operation is completed. For example, a substrate may be processed according to a first operation that deposits a first material on a surface of the substrate and a second operation that deposits a second material on the first material. The first and second operations may be subject to a time constraint that the second material must be deposited on the first material within a specific amount of time, otherwise the first material may begin to degrade and the substrate may become unusable (i.e., unusable) for manufacturing a finished product. A time constraint window refers to a specific amount of time for completing the operation that prompts the time constraint (called the start operation) and the amount of time that a subsequent operation (called the finish operation) must be completed after the start operation is completed. In some cases, one or more operations may be performed between the start operation and the finish operation.

[0003] In most cases, when a substrate arrives at a tool, an operation cannot be initiated on that substrate because the tool may be processing other substrates. Therefore, an operator of the manufacturing system (e.g., an industrial engineer, a process engineer, a systems engineer, etc.) schedules the operation to be performed at a particular time to satisfy a time constraint associated with the operation. For example, the operator may delay the execution of an operation on the substrate until each tool configured to perform an operation associated with the time constraint has the ability to perform the operation within the time constraint window.

[0004] In some cases, a completing operation of a first time constraint window may also be a starting operation of a second time constraint window. In such cases, an operator of the manufacturing system may schedule the starting operation of the first time constraint window to start at a particular time to satisfy the first time constraint of the first time constraint window and the second time constraint of the second time constraint window. In other cases, an operation may be a completing operation of both the first time constraint window and the second time constraint window. In such cases, an operator may schedule the starting operation of the first time constraint window and the second time constraint window to start at a particular time to satisfy the first time constraint of the first time constraint window and the second time constraint of the second time constraint window. Summary of the Invention [Problem to be solved by the invention]

[0005] As manufacturing systems become more complex, more operations are subject to time constraints. To schedule a substrate to be started at a start operation, an operator (e.g., using a computing system) considers all time constraints that may be prompted by the start operation. To consider all time constraints that may be prompted by the start operation, the operator considers the capabilities of each tool that can perform the start operation, the complete operation, and each operation in between. In some cases, the time constraint window that includes the start operation may represent a significant amount of time (e.g., 6 hours, 8 hours, 12 hours, 24 hours, etc.). The operator may have difficulty considering each time constraint and capability of each tool of the manufacturing system over a significant amount of time into the future. In some computing systems, this consideration may be classified as an NP-hard (nondeterministic polynomial time hard) problem. As such, the operator may not be able to schedule a substrate to be started at each start operation of a sequence of operations such that each time constraint is not satisfied. As a result, the substrate may violate the time constraint of the sequence of operations and become unavailable. Each substrate that becomes unavailable may reduce the throughput of the entire system and contribute to an increase in the latency of the entire system.

[0006] The following is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure, nor to delineate the scope or claims of particular embodiments of the disclosure. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later. [Means for solving the problem]

[0007] In one aspect of the disclosure, a method for training a software agent is provided. The method includes initializing the software agent to select an action to perform in a simulated environment associated with a manufacturing system and initiating a simulation of the selected action in the simulated environment. In response to pausing the simulation, the method further includes obtaining output data based on an environmental state associated with the simulation and updating the software agent to be configured to generate one or more dispatching decisions indicative of a time to begin processing one or more substrates in the manufacturing system based on the output data.

[0008] In another aspect of the disclosure, a method for time constraint management in a manufacturing system is provided. The method includes receiving a request to initiate a series of operations to be performed on a set of candidate substrates in the manufacturing system, the series of operations including one or more operations each having one or more time constraints. The method further includes obtaining current data regarding a current state of the manufacturing system and applying a software agent to the current data to determine a time to process the set of candidate substrates. The method further includes initiating the series of operations on the set of candidate substrates at the determined time.

[0009] A further aspect of the present disclosure includes an electronic device manufacturing system comprising a memory device and a processing device operatively coupled to the memory device to perform operations according to any aspect or implementation described herein.

[0010] A further aspect of the present disclosure includes a non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device operably coupled to a memory, perform operations according to any aspect or implementation described herein.

[0011] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary system architecture, in accordance with certain implementations. [Diagram 2] 1 illustrates an exemplary system for performing reinforcement learning to generate a software agent, according to certain embodiments. [Diagram 3] 1 is a flow diagram of a method for training a software agent, according to certain embodiments. [Figure 4] 1 is a top schematic diagram of an exemplary manufacturing system, in accordance with certain embodiments. [Diagram 5] 1 illustrates a sequence of operations subject to one or more time constraints, according to an embodiment of the present disclosure. [Figure 6] 1 is a flow diagram illustrating a method for initiating a sequence of operations based on a dispatching decision generated using a machine learning model, according to certain implementations. [Figure 7] FIG. 1 is a block diagram illustrating a computer system in accordance with certain embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Described herein are techniques directed to using reinforcement learning for time constraint management in manufacturing systems. In some processes, a series of operations may be performed at various stages of a manufacturing system. For example, a series of operations may be performed to deposit a coating (or multiple coatings) on a surface of a substrate and etch a three-dimensional pattern into the coating. In some cases, one or more of the series of operations may be time constrained. A time constraint may refer to a restriction or protocol in which after an operation is performed in a manufacturing system, a subsequent operation must be completed within a certain amount of time. For example, a manufacturing system may be time constrained such that after a coating is deposited on a surface of a substrate, an etching process must be performed on the substrate within a certain number of hours (e.g., 12 hours). If the time constraint is not met (e.g., if the etching process is not performed within a certain number of hours), the substrate may become defective and unusable.

[0014] Embodiments of the present disclosure are directed to using deep reinforcement learning to manage time constraints in a substrate manufacturing system. A processing device can receive a request to initiate an operation to be performed in the manufacturing system, where one or more operations are subject to a time constraint. The processing device can determine when to release certain substrates for processing given the time constraint so that these substrates can be successfully processed in the manufacturing system within a particular time period. For example, the processing device can identify a time during a sequence of operations when a set of candidate substrates should be processed in the substrate manufacturing system.

[0015] To identify the candidate set of substrates, the processing device can obtain data regarding the current state of the manufacturing equipment. The data can include current state data, sensor data, context data, task data, etc. For example, the current data can relate to one or more operations being performed on one or more substrates being processed, the number of substrates being processed on the manufacturing equipment at a particular time, the number of substrates in the manufacturing equipment's queue, the current useful life, setup data, a set of operations including individual processes performed in one or more manufacturing facilities in a production environment, sensor data, etc. The processing device can provide the data regarding the current state of the manufacturing equipment as input to an agent. The agent can include a software program that is capable of perceiving an environment, autonomously taking action to achieve one or more goals, and improving performance through learning.

[0016] An agent (also referred to herein as software or intelligent agent) may be used to generate dispatching decisions. The dispatching decisions may determine which action to perform at a given time in a production environment. Examples of dispatching decisions may include, but are not limited to, where a substrate should be processed next in a production environment, which substrate should be selected for an idle machine in a production environment, etc. Based on the dispatching decision data, a processing machine may initiate a series of operations on a set of candidate substrates at a particular time.

[0017] In some implementations, the dispatching decision may indicate at what time a candidate set of substrates should be processed (e.g., when a substrate to be started at a start operation should be scheduled). In other implementations, the dispatching decision may include decisions such as whether to start processing a batch with less than a permitted number of substrates, or whether to wait to start a batch until additional substrates are available so that a complete batch can be started. In still other implementations, the dispatching decision may include a decision to release a substrate waiting at a logical gating step so that the substrate can be dispatched for processing at a subsequent processing step. In some cases, to manage time constraints, the process flow includes a (logical) step of no processing before a step (called a gating step) that starts the time constraint. A lot (set of substrates) waits at the gating step until the system determines that it has the capacity to completely process the lot. If so, the lot is released from the gating step and made available for processing at the first operation in the time constraint. A software agent may control this gating step.

[0018] In some implementations, the software agent can be trained using deep reinforcement learning. Deep reinforcement learning combines artificial neural networks with a reinforcement learning framework to help the software agent learn how to reach a goal (e.g., deep reinforcement learning involves learning from existing knowledge and applying it to new data sets). In one example, during training, the software agent selects and simulates an action one time step into the future (in a simulated environment). The software agent then receives a new environment state and reward. The state-action-reward sequence is saved, and periodically, the reinforcement learning algorithm uses this experience to update the weights of the neural network, which represents the policy. The policy is used to select the next action. The policy update aims to maximize the cumulative reward over the time horizon. Once the learning curve stabilizes and the policy stops improving, the policy is saved and can be used for the current data associated with the manufacturing equipment.

[0019] Aspects and embodiments of the present disclosure address the shortcomings of existing technology by providing techniques for scheduling a substrate or set of substrates to begin at a start operation. A processing device can use trained software agents to determine candidate substrate sets for processing during a current or future time period (based on a sequence of operations). By applying the software agents, the processing device can obtain dispatching decisions indicating when to schedule a set of substrates for processing. By determining when to schedule a set of substrates, the processing device can schedule a set of substrates to begin at a sequence of operations within a time period such that few or no substrates violate the time constraints of the sequence of operations. As a result, few substrates, or nearly zero substrates, violate the time constraints of the sequence of operations, and a significant number of substrates processed in a manufacturing system contain no or few defects. Thus, the trained software agents can reduce latency violations while maintaining high throughput, as opposed to traditional heuristic solutions that may reduce throughput.

[0020] 1 is a block diagram illustrating a production environment 100 according to an aspect of the present disclosure. The production environment 100 can include multiple systems, such as, but not limited to, a production dispatcher system 103, manufacturing equipment 112 (e.g., manufacturing tools, automation equipment, etc.), client devices 114, a predictive system 116 (e.g., for generating predictive data such as dispatching decisions, for providing adaptation of models or agents, for using knowledge bases, etc.), and one or more computer integrated manufacturing (CIM) systems 101. Examples of the production environment 100 can include, but are not limited to, a manufacturing plant, a fulfillment center, etc. For brevity and simplicity, a manufacturing system is used as an example of the production environment 100 throughout this specification.

[0021] In some implementations, the production environment 100 may be a semiconductor manufacturing environment. In such implementations, the manufacturing tools 112 may perform a number of different operations related to the manufacture of semiconductor substrates. For example, the manufacturing tools 112 may perform sawing operations, cleaning operations, deposition operations, etching operations, testing operations, etc. Aspects of the present disclosure are described with respect to the manufacture of semiconductor substrates in a semiconductor manufacturing environment. However, it should be noted that embodiments of the present disclosure may be applied to other production environments 100 configured to manufacture or otherwise process lots other than semiconductor substrates. A lot may refer to a set of substrates.

[0022] The manufacturing equipment 112 can include sensors 126 configured to capture data of substrates being processed in the manufacturing equipment 112. In some implementations, the manufacturing equipment 112 and the sensors 126 can be part of a sensor system that includes a sensor server (e.g., a field service server (FSS) at the manufacturing facility) and a sensor identifier reader (e.g., a front-opening unified pod (FOUP) radio frequency identification (RFID) reader for the sensor system). In some implementations, the manufacturing equipment 112 can include or be operably coupled to a metrology equipment that includes a metrology server (e.g., metrology database, metrology folder, etc.) and a metrology identifier reader (e.g., a FOUP RFID reader for the metrology system).

[0023] The manufacturing tool 112 may perform operations according to a recipe or over a period of time to produce products such as electronic devices. The manufacturing tool 112 may include a process chamber. The manufacturing tool 112 may perform a process on a substrate (e.g., a wafer, etc.) in the process chamber. Examples of substrate processes include a deposition process to deposit one or more layers of a film on a surface of the substrate, an etching process to form a pattern on the surface of the substrate, etc. The manufacturing tool 122 may perform each process according to a process recipe. A process recipe defines a particular sequence of operations to be performed on the substrate during the process and may include one or more settings associated with each operation. For example, a deposition process recipe may include a temperature setting for the process chamber, a pressure setting for the process chamber, flow rate settings for precursors of materials included in the film to be deposited on the substrate surface, etc.

[0024] In some implementations, the sensors 126 provide sensor data (e.g., sensor values, characteristics, trace data) associated with the manufacturing equipment 112 (e.g., associated with the production of a corresponding product, such as a wafer, by the manufacturing equipment 112). The manufacturing equipment 112 can produce a product according to a recipe or by performing an operation over a period of time. The sensor data received over a period of time (e.g., corresponding to at least a portion of a recipe or operation) can be referred to as trace data received over time from the different sensors 126 (e.g., historical trace data, current trace data, etc.). The sensor data can include one or more values ​​of temperature (e.g., heater temperature), spacing (SP), pressure, high frequency radio frequency (HFRF), electrostatic chuck (ESC) voltage, current, material flow, power, voltage, etc. The sensor data can be associated with or indicative of hardware parameters, such as settings or components (e.g., size, type, etc.) of the manufacturing equipment 124, or manufacturing parameters, such as process parameters of the manufacturing equipment 112. The sensor data can be provided while the manufacturing equipment 112 is performing a manufacturing process (e.g., equipment readings as the product is processed). The sensor data may vary from board to board.

[0025] The CIM 101, the production dispatcher system 103, the manufacturing equipment 112, the client devices 114, the forecasting system 116, and the data stores 140, 150 can be coupled to one another via a network 120. The network 120 can include one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet networks), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof. The CIM system 101, the production dispatcher system 103, and the forecasting system 116 can be hosted individually or together in any combination by any type of machine, including a server computer, a gateway computer, a desktop computer, a laptop computer, a tablet computer, a notebook computer, a PDA (personal digital assistant), a mobile communication device, a mobile phone, a smartphone, a handheld computer, or a similar computing device. In some implementations, the prediction system 116 is part of a server hosted on a machine.

[0026] The data stores 140, 150 can be memory (e.g., random access memory), drives (e.g., hard drives, flash drives), database systems, or another type of component or device capable of storing data. The data stores 140, 150 can include multiple storage components (e.g., multiple drives or multiple databases) that can span multiple computing devices (e.g., multiple server computers).

[0027] The data store 140 can store data associated with processing of substrates in the manufacturing tool 112. For example, the data store 140 can store data (referred to as process data) collected by the sensors 126 of the manufacturing tool 112 before, during, or after the substrate process. The process data can refer to historical process data (e.g., process data generated for a previous substrate processed in the manufacturing system) and / or current process data (e.g., process data generated for a current substrate processed in the manufacturing system). The data store can also store spectral or non-spectral data associated with a portion of a substrate processed in the manufacturing tool 112. The spectral data can include historical spectral data and / or current spectral data.

[0028] The data store 140 may also store context data associated with one or more substrates processed in the manufacturing system. The context data may include recipe name, recipe step number, preventive maintenance indicator, operator, etc. The context data may refer to historical context data (e.g., context data associated with a previous process performed on a previous substrate) and / or current process data (e.g., context data associated with a current or future process performed on a previous substrate). The context data may further include identifying a sensor associated with a particular subsystem of a process chamber.

[0029] The data store 140 may also store task data. The task data may include one or more series of operations to be performed on a substrate during a deposition process and may include one or more settings associated with each operation. For example, task data for a deposition process may include a temperature setting for a process chamber, a pressure setting for the process chamber, flow rate settings for precursors of a material of a film deposited on the substrate, etc. In another example, the task data may include a control pressure at a pressure point defined relative to a flow rate value. The task data may refer to past task data (e.g., task data associated with a previous process performed on a previous substrate) and / or current task data (e.g., task data associated with a current or future process to be performed on a substrate).

[0030] In some implementations, data store 140 can be configured to store data that is not accessible to users of the manufacturing system. For example, process data, spectral data, context data, etc. acquired for substrates being processed in the manufacturing system are not accessible to users (e.g., operators) of the manufacturing system. In some implementations, all data stored in data store 140 can be inaccessible to users of the manufacturing system. In other or similar implementations, some of the data stored in data store 140 can be inaccessible to users, while other portions of the data stored in data store 140 can be accessible to users. In some implementations, one or more portions of the data stored in data store 140 can be encrypted using an encryption mechanism not known to the user (e.g., the data is encrypted using a private encryption key). In other or similar implementations, data store 140 can include multiple data stores, where data that is not accessible to users is stored in one or more first data stores and data that is accessible to users is stored in one or more second data stores.

[0031] The data store 150 stores dispatching rules 151, status data 153, and user data 155. The dispatching rules 151 can be logic executable by the production dispatcher system 103. In some implementations, the dispatching rules 151 can be defined by a user (e.g., an industrial engineer, a process engineer, a systems engineer, etc.). Examples of dispatching rules 151 can include, but are not limited to, selecting the highest priority substrate to work on next, selecting a substrate that uses the same setup as the tool is currently configured for, packaging the item when the order is complete, shipping the item when packaging is complete, etc. Each dispatching rule 151 can be associated with a number of data operations to implement the corresponding dispatching rule 151. Examples of data operations can include, but are not limited to, importing data, compressing data, indexing data, filtering data, performing mathematical functions on data, etc.

[0032] The state data 153 may include a state of the manufacturing equipment 112 (e.g., operating temperature, operating pressure, number of substrates being processed on the manufacturing equipment, number of substrates in the manufacturing equipment's queue at a particular time, current useful life, setup data, a set of operations including individual processes performed at one or more manufacturing facilities of the production environment, etc.). The state data 153 may be generated by the manufacturing equipment 112 during operation of the production environment 100 and stored in the data store 150. The state data 153 may include one or more of current state data, past state data, and perturbation state data. The current state data may include data regarding a current state of the manufacturing equipment 112 (e.g., current operating temperature, current operating pressure, current number of substrates being processed on the manufacturing equipment, etc.). The past state data may include data regarding a past state of the manufacturing equipment 112 (e.g., past operating temperature at a particular time, past operating pressure at a particular time, past number of substrates being processed on the manufacturing equipment at a particular time, etc.). The perturbation state data may include changed state data. In particular, the perturbed state data can include current or past state data that has one or more parameters altered or distorted. The one or more parameters can be altered based on user input, a particular percentage, a particular value, randomly altered, etc. For example, the perturbed state data can include a number of past substrates that were processed on the manufacturing tool at a particular time, increased or decreased by a predetermined value of two substrates. In another example, the perturbed state data can include a number of past sets of substrates that were processed on the manufacturing tool at a particular time, increased or decreased by a set of random numbers, e.g., 1-10. In some implementations, the state data 153 can include or be generated from data stored in the data store 140. For example, the state data 153 can include or be generated from sensor data, context data, task data, etc.

[0033] In some implementations, state data may refer to data regarding environmental conditions of a simulated environment (e.g., environment 204). Environmental state data may include manufacturing equipment characteristics (e.g., step processing times, wait time constraints, etc.), manufacturing equipment observations (e.g., number of substrates or lots processed per step, number of lots processed per station, etc.), wait time observations (e.g., number of successful lots processed, number of violating lots, number of lots in process, etc.), capacity observations (e.g., estimates of time to complete all work in progress (WIP)). Environmental state features may be normalized to values ​​of [0,1] and concatenated into a single observation vector.

[0034] User data 155 may include data provided by a user (e.g., an operator, a process engineer, an industrial engineer, a systems engineer, etc.) of production environment 100. In some implementations, user data 155 may be provided via client device 114.

[0035] User devices 114 may include computing devices such as personal computers (PCs), laptops, mobile phones, smart phones, tablet computers, netbook computers, network-connected televisions, etc. In some implementations, user devices 114 may provide information to users of production environment 100 (e.g., operators, industrial engineers, process engineers, systems engineers, etc.) via one or more graphical user interfaces (GUIs).

[0036] Examples of CIM systems 101 may include, but are not limited to, manufacturing execution systems (MES), enterprise resource planning (ERP), production planning and control (PPC), computer-aided systems (e.g., design, engineering, manufacturing, process planning, quality assurance), computer numerically controlled machines, direct numerically controlled machines, controllers, and the like.

[0037] In some implementations, prediction system 116 includes a prediction server 118 and a server machine 180. Prediction server 118 and server machine 180 may each include one or more computing devices, such as a rack mount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a graphic processing unit (GPU), an accelerator application specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc.

[0038] The prediction system 116 can train a software agent 190 (e.g., an intelligent agent). A software agent is a computer program that acts in a proxy relationship on behalf of a user or other program. In some implementations, the software agent 190 can be trained using reinforcement learning, deep reinforcement learning, and the like. Reinforcement learning is a class of algorithms applicable to sequential decision-making tasks. In particular, reinforcement learning is a process by which a software agent learns to make decisions through trial and error.

[0039] In some embodiments, training the software agent can include using deep reinforcement learning. Deep reinforcement learning combines artificial neural networks with a reinforcement learning framework to help the software agent learn how to reach a goal. In particular, deep reinforcement learning integrates function approximation and goal optimization to map states and actions to the rewards they yield. Deep reinforcement learning involves learning from existing knowledge and applying it to new data sets, while reinforcement learning can involve dynamically learning in a trial-and-error manner to maximize results. In one embodiment, a neighborhood policy optimization (PPO) algorithm can be used to train the software agent 190. The PPO algorithm is a deep RL algorithm that uses a policy gradient method to train a probabilistic policy in an on-policy manner. The PPO algorithm also utilizes an actor-critic method. More details regarding training the software agent 190 using deep reinforcement learning are described below in FIG. 2 and FIG. 3.

[0040] Deep learning is a class of machine learning algorithms that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks include hierarchical structures, with different layers learning different levels of representations corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into slightly more abstract and complex representations. Notably, the deep learning process can teach itself which features to best place at which levels. The "deep" in "deep learning" refers to the number of layers through which data is transformed. More precisely, deep learning systems have a significant confidence assignment path (CAP) depth. A CAP is a chain of transformations from input to output. A CAP describes the potential causal relationship between the input and the output. For feedforward neural networks, the CAP depth can be the depth of the network, which can be the number of hidden layers + 1. For recurrent neural networks, where a signal can propagate through a layer more than once, the depth of CAP is potentially unlimited.

[0041] Training a neural network can be accomplished in a supervised learning manner, which involves feeding a training dataset of labeled inputs through the network, observing its output, defining an error (by measuring the difference between the output and the label value), and adjusting the network weights across all layers and nodes of the network to minimize the error, using techniques such as deep gradient descent and backpropagation. In many applications, repeating this process across many labeled inputs in the training dataset results in a network that can generate the correct output when presented with inputs different from those present in the training dataset.

[0042] In some implementations, training of the neural network can be accomplished using reinforcement learning. Reinforcement learning differs from supervised learning in that it does not require the presentation of labeled input / output pairs and does not require explicit correction of suboptimal actions. The focus of reinforcement learning can be on finding a balance between exploring unknown territory and leveraging current knowledge. Partially supervised reinforcement algorithms can combine the advantages of supervised and RL algorithms.

[0043] The server machine 180 may include a training engine 182. An engine may refer to hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, a processing unit, etc.), software (e.g., instructions executed on a processing unit, a general-purpose computer system, or a dedicated machine), firmware, microcode, or a combination thereof. The training engine 182 may train one or more software agents 190. The software agents 190 may be created by the training engine 182 using training data (also referred to herein as a training set), including a simulation environment, rewards, actions, states (e.g., observations), etc.

[0044] To perform the training, the processing logic can input a training data set into one or more simulation environments. Prior to inputting the first input into the simulation environments, the software agent can be initialized. The processing logic trains the software agent based on actions provided to the simulation environments and rewards and observations obtained from the simulation environments (based on the simulation state). The processing logic can pause the simulation, and the software agent processes the obtained observations (e.g., state data) and reward data and selects new actions to input into the simulation. The simulation can then be resumed, which can be performed iteratively until the simulation is complete. The software agent can be trained on multiple simulations. Once trained, the software agent can be applied to current state data of the manufacturing equipment and generate an output indicative of one or more predictions or inferences. For example, the output predictions or inferences can include whether a particular candidate set of substrates can begin within a given amount of time (e.g., the next 15 minutes) with a time-bound constraint, when one or more substrates should be released for processing, etc.

[0045] After one or more training rounds, the processing logic can determine whether a stopping criterion is met. The stopping criterion can be a target level of accuracy, a target number of processed images from the training dataset, a target amount of change to a parameter over one or more previous data points, combinations thereof, and / or other criteria. In one embodiment, the stopping criterion is met when at least a minimum number of data points have been processed and at least a threshold accuracy has been achieved. The threshold accuracy can be, for example, 70%, 80% or 90% accuracy. In one embodiment, the stopping criterion is met when the accuracy of the machine learning model stops improving. If the stopping criterion is not met, further training is performed. If the stopping criterion is met, the training can be completed. Once the machine learning model is trained, the model can be tested using a reserved portion of the training dataset.

[0046] Once one or more trained software agents 190 are generated, they can be stored in the prediction server 118 as a prediction component 119 or as a component of the prediction component 119.

[0047] As described in more detail below, the prediction server 118 includes a prediction component 119 that can run trained software agents 190 against current state data to provide predictive data indicating when one or more substrates should be released for processing, how many substrates can be successfully processed in the manufacturing system according to a sequence of operations having one or more time constraints, etc., as described in more detail below.

[0048] It should be noted that in some other implementations, the functionality of server machine 180 as well as the functionality of prediction server 118 may be provided by fewer machines. For example, in some implementations, server machine 180 and prediction server 118 may be combined into a single machine.

[0049] In general, functions described in one embodiment as being performed by server machine 180 and / or prediction server 118 may also be performed on client device 114. Additionally, functions attributed to a particular component may be performed by different components or multiple components working in conjunction.

[0050] In an embodiment, a "user" may be represented as a single individual. However, other embodiments of the present disclosure encompass a "user" being an entity controlled by multiple users and / or automated sources. For example, a collection of individual users aggregated as a group of administrators may be considered a "user."

[0051] The production dispatcher system 103 can make dispatching decisions for the production environment 100. Dispatching decisions determine what action to perform at a given time in the production environment 100. Dispatching often includes decisions such as whether to start processing a batch, whether to start processing a batch with less than an acceptable number of substrates, or whether to wait to start a batch until additional substrates are available so that a complete batch can be started. Examples of dispatching decisions can include, but are not limited to, where a substrate should be processed next in the production environment, which substrate should be selected for an idle device in the production environment, and the like. In some implementations, the production dispatcher system 103 can make dispatching decisions using forecast data generated by the forecast component 119. In some implementations, the production dispatcher system 103 can make dispatching decisions using one or more dispatching rules 151 stored in the data store 150.

[0052] In some cases, a manufacturing process may include hundreds of operations performed by manufacturing equipment 112 (e.g., tools or automated equipment) in the production environment 100. Often, one or more operations may be subject to a time constraint. As previously mentioned, a time constraint refers to a specific amount of time after an operation is completed that a subsequent operation must be completed. For example, after a first material is deposited on a surface of a substrate, a second material is to be deposited on the first material within a specific amount of time after the deposition of the first material. If the second coating is not deposited on the first material within the specific amount of time, the first material may begin to degrade and the substrate may become unusable. A time constraint window refers to the amount of time to complete a first operation (called a start operation) and the specific amount of time that a second operation (called a finish operation) must be completed. In some implementations, one or more operations performed between the start and finish operations are also associated with a time constraint window. Following the previous example, a time constraint window can refer to a first amount of time for depositing a first material on a surface of a substrate and a particular amount of time in which a second material must be deposited on the first material. Operations can be subject to one or more time constraints. In some implementations, a completing operation of a first time constraint window can also be a starting operation of a second time constraint window.

[0053] 2 illustrates an exemplary system 200 for performing reinforcement learning to generate a software agent, according to certain embodiments of the present disclosure. The exemplary system 200 includes a software agent 202 and a simulated environment 204 (e.g., a simulator). The agent 202 takes actions that affect the environment 204 and change its state (e.g., environmental state). The environmental state is a representation of the current environment in which the agent resides. This state can be observed by the agent 202 and contains all the relevant information about the environment that the agent 202 needs to know to make a decision (e.g., perform an action). Following each action, the agent 202 transitions to the next environmental state and receives a reward.

[0054] The agent 202 can use one or more machine learning models 240. The machine learning models 240 can be, for example, deep neural networks (e.g., convolutional neural networks, transformers, graph neural networks, etc.) or decision trees. The machine learning models 240 can represent policies (e.g., solution policies). A policy can be a strategy of actions that promises the highest reward in the long term.

[0055] The agent 202 may receive rewards for taking control that leads to a successful environmental state. Rewards may be immediate, such as receiving points for each step taken in the right direction, or delayed, such as receiving points at the end of an episode if a goal is reached. An episode may refer to a sequence of environmental states, actions, and rewards, terminating in a final environmental state. In an illustrative example, each episode (or experiment) may include 100 time steps, and each time step may take 100 minutes. At each time step, the agent 202 may take a single action. Following the action, the agent 202 receives observations (e.g., environmental state data) that reflect the state of the environment 204 at the end of the time step. An episode ends when 100 time steps have passed or when a predetermined number of lots (e.g., 10 lots) have completed a route, whichever occurs first.

[0056] In some implementations, the exemplary system 200 uses a form of Markov decision process (MDP) in which the agent 202 attempts to optimize a function in its environment 204. An MDP defines an environment state space S (state s∈S), an action space A (a∈A), a transition function T: S×A→S, and a reward function

[0057]

number

[0058] In some implementations, latency management can be modeled as a discrete-time, finite-horizon MDP, which is a tuple M=(S, A, P, R, ρ 0 ,T), where S is the set of environmental states, A is the set of actions, P:S×A×S→R+ is the transition probability distribution, R:S×A→R is the reward function, ρ 0 :S→[0,1] is the initial environment state distribution and T is the time horizon. A solution policy can be a probability distribution π:S×A→[0,1] that maps the environment states to actions. To find a solution policy, the agent 202 calculates the expected return

[0059]

number

[0060] During training, the agent 202 takes an action. The environment 204 applies the action, simulating one time step into the future. The agent 202 then receives new environment state data and a new reward. The state-action-reward sequence is stored, and periodically, a reinforcement learning algorithm uses this experience to update the weights of a neural network (e.g., machine learning model 240) that represents a policy. The policy is used to select the next action. The policy update aims to maximize the cumulative reward over a time horizon. Once the learning curve stabilizes and the policy stops improving, the processing logic (e.g., training engine 182) can store the policy and use it to test the performance of the software agent 202 in one or more environments.

[0061] Environmental state data (e.g., data regarding the state of the environment 204) may include manufacturing equipment characteristics (e.g., step processing times, wait time constraints, etc.), manufacturing equipment observations (e.g., number of substrates or lots processed per step, number of lots processed per station, etc.), wait time observations (e.g., number of successful lots processed, number of violating lots, number of lots in process, etc.), capacity observations (e.g., estimates of time to complete all work in progress (WIP)), quantities of lots or substrates waiting to process various steps and / or waiting to start various time constraints, etc. The state features may be normalized to values ​​of [0,1] and concatenated into a single observation vector.

[0062] At each time step, the agent 202 can decide to release or not release a lot. The agent 202 can release one lot of N part types (or a lot waiting for a processing or gating step). Thus, the agent 202 can choose a discrete action from 0 to N. Selecting action 0 will not release the lot, action a will not release the lot, and action b will not release the lot. i Select type Part i The agent 202 can also select actions including releasing multiple lots, potentially of different part types or steps.

[0063] The reward structure can be configured to encourage the agent 202 to optimize the makespan (e.g., the time difference between the start and finish of a series of jobs or tasks) and the number of successful lots while minimizing the number of wait time violations. The structured reward can also be configured to encourage the agent 202 to maximize the throughput of a manufacturing device.

[0064] 3 is a flow diagram of a method 300 for training a software agent according to an aspect of the disclosure. The method 300 is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (such as running on a general-purpose computer system or a dedicated machine), firmware, or some combination thereof. In one implementation, the method 300 may be performed by a computer system such as the computer system architecture 100 of FIG. 1. In other or similar implementations, one or more operations of the method 300 may be performed by one or more other machines not shown in the figure. In some aspects, one or more operations of the method 300 may be performed by the server machine 180 and / or the prediction server 118.

[0065] For ease of explanation, the method is shown and described as a series of acts. However, acts according to the present disclosure can be performed in various orders and / or simultaneously, as well as with other acts not shown and described herein. Furthermore, not all acts shown may be performed to implement a method in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that a method may alternatively be represented as a series of interrelated states via a state diagram or events. Furthermore, it should be appreciated that the methods disclosed herein can be stored in an article of manufacture to facilitate transporting and transferring such methods to a computing device. The term article of manufacture, as used herein, is intended to encompass a computer program accessible from any computer-readable device or storage medium.

[0066] In operation 310, processing logic initializes a software agent. In some implementations, the software agent may have access to environmental state data and / or state data (e.g., data associated with operations related to the fabrication of semiconductor substrates, such as historical state data, current state data, perturbation state data, etc.).

[0067] In operation 312, processing logic executes one or more of the simulations. The one or more simulations may be executed in a simulation environment (e.g., environment 204). In some implementations, the simulation may include simulating an action (e.g., one time step into the future). In some implementations, processing logic may determine a particular time period during which a training set of operations should be performed in the manufacturing system. In some implementations, the training set of operations may be the sequence of operations illustrated by FIG. 5. The particular time period may be a simulation condition according to the implementations described above.

[0068] In some embodiments, the simulation may be performed in response to the software agent selecting action data. The action data may include a set of possible moves, actions, or operations that the software agent may take. In some embodiments, the action may include not releasing the lot, releasing a particular lot, releasing the lot for a particular process chamber, releasing the lot during a particular time period, etc. In some embodiments, the action may include determining a training set of substrates to be processed during a training set of operations. The training set of candidate substrates and the training set of operations are determined using state data, operator input, a predefined set of rules (e.g., one or more predefined sets of substrates, one or more predefined sequences of operations, etc.), random input, or any combination thereof.

[0069] At operation 314, processing logic pauses the simulation to obtain output data. In some implementations, the output data may include new environmental state data and reward data based on the current environmental state.

[0070] At operation 316, processing logic updates the software agent based on the output data (e.g., the new environmental state data and the new reward data). The new reward data may include feedback data that measures the success or failure of an action in a given state.

[0071] At operation 318, processing logic generates, by the software agent, new action (eg, action data) data based on the new state data.

[0072] At operation 320, processing logic restarts the simulation using the new action data. For example, processing logic may simulate a new action in the environment.

[0073] Processing logic may perform operations 312-316 until the simulation or set of simulations is complete. Processing logic may perform operation 300 until training of the software agent is complete. In some implementations, the output data indicates the number of candidate substrates that were successfully processed and reached the end of the time period during each operation in the simulated set of operations.

[0074] It should be noted that while in some implementations the sufficiency of training may be determined solely based on the amount of training data or the number of updates to the software agent, in some other implementations the sufficiency of training may be determined based on one or more other criteria (e.g., a measure of the diversity of the training examples, etc.).

[0075] After operation 318, a software agent can be used to generate predictive data (e.g., dispatching decisions) based on the current state data. In some implementations, the predictive data can include one or more dispatching decisions. For example, a machine learning model can receive as input the current state data and output a dispatching decision. As described above, the dispatching decision determines what action to take at a given time in the production environment 100. Dispatching can include decisions such as whether to start processing a batch with less than an acceptable number of substrates or whether to wait to start a batch until additional substrates are available so that a full batch can be started. Examples of dispatching decisions can include, but are not limited to, where a substrate should be processed next in the production environment, which substrate should be selected for an idle device in the production environment, etc.

[0076] 4 is a top schematic view of an exemplary manufacturing system 400 according to an embodiment of the present disclosure. The manufacturing system 400 is capable of performing one or more processes on a substrate 402. The substrate 402 can be any suitably rigid, planar article of fixed dimensions, such as, for example, a silicon-containing disk or wafer, a patterned wafer, a glass plate, etc., suitable for fabricating electronic devices or circuit components thereon.

[0077] The manufacturing system 400 can include a process tool 404 and a factory interface 406 coupled to the process tool 404. The process tool 404 can include a housing 408 having a transfer chamber 410 therein. The transfer chamber 410 can include one or more process chambers (also referred to as processing chambers) 414, 416, 418 disposed therearound and coupled thereto. The process chambers 414, 416, 418 can be coupled to the transfer chamber 410 via respective ports, such as slit valves. The transfer chamber 410 can also include a transfer chamber robot 412 configured to transfer the substrate 402 between the process chambers 414, 416, 418, a load lock 420, or the like. The transfer chamber robot 412 can include one or more arms, each arm including one or more end effectors at the end of each arm. The end effectors can be configured to handle a particular object, such as a wafer, a sensor disk, a sensor tool, or the like.

[0078] The process chambers 414, 416, 418 can be adapted to perform any number of processes on the substrate 402. The same or different substrate processes can be performed in each processing chamber 414, 416, 418. The substrate processes can include atomic layer deposition (ALD), physical vapor deposition (PVD), chemical vapor deposition (CVD), etching, annealing, curing, pre-cleaning, metal or metal oxide removal, and the like. Other processes can be performed on the substrate in the process chambers. Each of the process chambers 414, 416, 418 can include one or more sensors configured to capture data of the substrate 402 before, after, or during the substrate process. For example, the one or more sensors can be configured to capture spectral and / or non-spectral data of a portion of the substrate 402 during the substrate process. In other or similar implementations, the one or more sensors can be configured to capture data associated with the environment in the process chambers 414, 416, 418 before, after, or during the substrate process. For example, the one or more sensors can be configured to capture data associated with the temperature, pressure, gas concentrations, etc. of the environments within the process chambers 414, 416, 418 during substrate processing.

[0079] A load lock 420 may also be coupled to the housing 408 and the transfer chamber 410. The load lock 420 may be configured to interface and be coupled to the transfer chamber 410 on one side and to the factory interface 406. The load lock 420 may have an environmentally controlled atmosphere that may be changed, in some implementations, from a vacuum environment (where substrates may be transferred to and from the transfer chamber 410) to an inert gas environment at or near atmospheric pressure (where substrates may be transferred to and from the factory interface 406). The factory interface 406 may be any suitable enclosure, such as, for example, a front-end equipment module (EFEM). The factory interface 406 may be configured to receive substrates 402 from substrate carriers 422 (e.g., front-opening unified pods (FOUPs)) docked to various load ports 424 of the factory interface 406. A factory interface robot 426 (shown in dotted lines) may be configured to transfer substrates 402 between the carriers (also called containers) 422 and the load lock 420. The carrier 422 may be a substrate storage carrier or a replacement part storage carrier.

[0080] The fabrication system 400 may also be connected to client devices (not shown) configured to provide information regarding the fabrication system 400 to a user (e.g., an operator). In some implementations, the client devices may provide information to a user of the fabrication system 400 via one or more graphical user interfaces (GUIs). For example, the client devices may provide information via the GUI regarding a target thickness profile of a film to be deposited on a surface of the substrate 402 during a deposition process performed in the process chambers 414, 416, 418. The client devices may also provide information regarding modifications to the process recipe given a respective set of deposition settings that are predicted to correspond to the target profile according to implementations described herein.

[0081] The manufacturing system 400 may also include a system controller 428. The system controller 428 may be and / or may include a computing device such as a personal computer, a server computer, a programmable logic controller (PLC), a microcontroller, etc. The system controller 428 may include one or more processing devices, which may be a general-purpose processing device such as a microprocessor, a central processing unit, etc. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or combinations of instruction sets. The processing device may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The system controller 428 may include a data storage device (e.g., one or more disk drives and / or solid state drives), a main memory, a static memory, a network interface, and / or other components. The system controller 428 may execute instructions to perform any one or more of the methodologies and / or implementations described herein. In some implementations, the system controller 428 may execute instructions to perform one or more operations in the manufacturing system 400 according to a process recipe. The instructions may be stored in a computer-readable storage medium, which may include (during execution of the instructions) a main memory, a static memory, a secondary storage device, and / or a processor.

[0082] The system controller 428 may receive data from sensors contained on or within various portions of the manufacturing system 400 (e.g., processing chambers 414, 416, 418, transfer chamber 410, load lock 420, etc.). In some implementations, the data received by the system controller 428 may include spectral and / or non-spectral data for a portion of the substrate 402. In other or similar implementations, the data received by the system controller 428 may include data associated with processing the substrate 402 in the processing chambers 414, 416, 418, as previously described. For purposes of this description, the system controller 428 is described as receiving data from sensors contained within the process chambers 414, 416, 418. However, the system controller 428 may receive data from any portion of the manufacturing system 400 and use the data received from that portion in accordance with the implementations described herein. In an illustrative example, the system controller 428 may receive data from one or more sensors in the process chambers 414, 416, 418 before, after, or during substrate processing in the process chambers 414, 416, 418. The data received from the sensors in various portions of the manufacturing system 400 may be stored in a data store 450. The data store 450 may be included as a component within the system controller 428 or may be a separate component from the system controller 428. In some implementations, the data store 450 may be the data stores 140, 150 described with respect to FIG. 1.

[0083] FIG. 5 illustrates a series of operations 500 subject to one or more time constraints, according to an embodiment of the present disclosure. Each operation 510 in the training set of operations may correspond to an individual process performed in one or more manufacturing facilities of the production environment, such as the manufacturing equipment 112 (e.g., tools or automated equipment) of the production environment 100. In some embodiments, each of the series of operations 500 may be sequential operations (e.g., each operation 510 is performed according to a particular order). In some embodiments, each operation 510 may correspond to an individual process performed in a front-end manufacturing facility, including, but not limited to, photolithography, deposition, etching, cleaning, ion implantation, chemical and mechanical polishing, etc. In other or similar embodiments, each operation may correspond to an individual process performed in a back-end manufacturing facility, including, but not limited to, dicing a completed wafer into individual semiconductor dies, testing, assembly, packaging, etc.

[0084] As previously mentioned, one or more operations 510 may be subject to time constraints. For example, operation 2 may be a first deposition operation that deposits a first material on a surface of a substrate, and operation 3 may be a second deposition operation that deposits a second material on the first material. Operations 2 and 3 may be subject to a first time constraint that the second material should be deposited on the first material within a certain amount of time (e.g., 6 hours) after the first material is deposited on the surface of the substrate. The amount of time for the manufacturing tool 112 to perform operations 2 and 3 may correspond to a time constraint window 512. The time constraint window 512 may include a first amount of time to complete the start operation (i.e., operation 510 starting the time constraint window 512) and a certain amount of time for the manufacturing tool 112 to complete the finish operation (i.e., operation 510 completing the time constraint window 512). According to the previous example, operation 2 should be started on a substrate in the manufacturing tool 112 such that operations 2 and 3 are completed on the substrate within the first time constraint window 512A.

[0085] In some implementations, a completed operation of a time constraint window 512 may be a starting operation of another time constraint window 512. For example, operation 3 may be a second deposition operation and operation 6 may be an etching operation. Operations 3, 4, 5, and 6 may be time constrained such that operation 6 should etch the second material within a certain amount of time (e.g., 12 hours) after operation 3 deposits the second material. The second time constraint window 512B may include an amount of time for operation 3 to deposit the second material and a certain amount of time for operation 6 to complete. Operation 3 should be started in the fabrication tool 112 such that operations 3, 4, 5, and 6 are completed within the second time constraint window 512B. Following the previous example, operation 3 may be time constrained with operation 2. Thus, operation 2 should be started on the substrate such that operations 2 and 3 are completed on the substrate within the first time constraint window 512A and operations 3, 4, 5, and 6 are completed within the second time constraint window 512B. The first time constraint window 512A and the second time constraint window 512B together are referred to as a cascaded time constraint window.

[0086] In some implementations, operation 510 may be subject to more than one time constraint. For example, operations 6, 7, 8, 9, and 10 may be subject to a first time constraint that operation 10 should be completed within a certain amount of time after operation 6 is completed. The third time constraint window 512C may include an amount of time for performing operation 6 and a certain amount of time for completing operation 10. Operations 9 and 10 may also be subject to a second time constraint that operation 10 should be completed within a certain amount of time after operation 9 is completed. The fourth time constraint window 512D may include an amount of time for completing operation 9 and a certain amount of time for completing operation 10. Thus, operation 6 should be started such that operations 6, 7, 8, 9, and 10 are completed within the third time constraint window 512D, and operations 9 and 10 are completed within the fourth time constraint window. The third time constraint window 512C and the fourth time constraint window 512 together are referred to as nested time constraint windows.

[0087] 6 is a flow diagram of a method 600 for initiating a sequence of operations based on a dispatching decision generated using a software agent, according to an aspect of the disclosure. The method 600 is performed by processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (such as running on a general-purpose computer system or a dedicated machine), firmware, or some combination thereof. In one implementation, the method 600 may be performed by a computer system, such as the computer system architecture 100 of FIG. 1. In other or similar implementations, one or more operations of the method 600 may be performed by one or more other machines not shown. In some aspects, one or more operations of the method 600 may be performed by the server machine 180, the prediction server 118, the CIM system 101, and / or the production dispatcher system 103.

[0088] In operation 610, processing logic receives a request to initiate a series of operations to be performed on a manufacturing system. In some implementations, the manufacturing system may be production environment 100 of FIG. 1. In some implementations, the request may be a request to initiate a series of operations to be performed on a manufacturing system at a particular time. For example, the request may be a request to initiate a series of operations at 8:00 PM. In some implementations, the request may be a request to initiate a series of operations on a candidate set of substrates. In some implementations, the request may be a request for a dispatching decision regarding the candidate set of substrates. For example, the request may request the next available time to initiate a series of operations on a candidate set of substrates that does not result in a time constraint issue.

[0089] In operation 612, the processing logic obtains current data regarding a current state of the manufacturing equipment. In some implementations, the current data may include current state data, sensor data, context data, task data, and the like. In some implementations, the current data may include a number of substrates being processed on the manufacturing equipment at a particular time, a number of substrates in a queue for the manufacturing equipment, a current useful life, setup data, a set of operations including individual processes performed in one or more manufacturing facilities in a production environment, and the like. In some implementations, the current data may relate to one or more operations being performed on one or more substrates being processed. For example, the operations may include a deposition process being performed in a process chamber to deposit one or more layers of films on a surface of the substrate, an etching process being performed on one or more layers of films on the surface of the substrate, and the like. The operations may be performed according to a recipe. The sensor data may include one or more values ​​of temperature (e.g., heater temperature), clearance, pressure, radio frequency radio frequency, electrostatic chuck voltage, current, material flow, power, voltage, and the like. The sensor data may be associated with or indicative of hardware parameters, such as settings or components (eg, size, type, etc.) of the manufacturing equipment 112, or manufacturing parameters, such as process parameters of the manufacturing equipment 112.

[0090] At operation 614, processing logic applies a software agent (e.g., agent 190) to the obtained current data. The software agent can be used to generate predictive data including one or more dispatching decisions.

[0091] In operation 616, processing logic initiates a series of operations in the manufacturing system to process the set of candidate substrates for the specified time period.

[0092] In some embodiments, the predictive data includes one or more dispatching decisions. A dispatching decision determines which action to perform at a given time in the production environment 100. In some embodiments, a dispatching decision may include a set of candidate substrates and a specified time period. A software agent may generate the predictive data.

[0093] FIG. 7 is a block diagram illustrating a computer system 700, according to certain implementations. In some implementations, the computer system 700 can be connected to other computer systems (e.g., via a network such as a local area network (LAN), an intranet, an extranet, or the Internet). The computer system 700 can operate in the capacity of a server or a client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 700 can be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Additionally, the term "computer" is intended to include any collection of computers that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methods described herein.

[0094] In a further aspect, the computer system 700 may include a processing unit 702, a volatile memory 704 (e.g., random access memory (RAM)), a non-volatile memory 706 (e.g., read only memory (ROM) or electrically erasable programmable ROM (EEPROM)), and a data storage device 716, which may communicate with each other via a bus 708.

[0095] The processing unit 702 may be provided by one or more processors, such as a general-purpose processor (e.g., a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor implementing other types of instruction sets, or a combination of multiple types of instruction sets, etc.) or a special-purpose processor (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor, etc.).

[0096] The computer system 700 may further include a network interface device 722 (e.g., coupled to a network 774). The computer system 700 may also include a video display unit 710 (e.g., an LCD), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), and a signal generating device 720.

[0097] In some implementations, the data storage device 716 may include a non-transitory computer-readable storage medium 724 capable of storing instructions 726 encoding any one or more of the methods or functions described herein, including instructions encoding the components of FIG. 1 (e.g., the prediction component 119, the time constraint simulation module 107, etc.), and instructions for implementing the methods described herein.

[0098] The instructions 726 may reside, completely or partially, within the volatile memory 704 and / or within the processing unit 702 during execution thereof by the computer system 700, and thus, the volatile memory 704 and the processing unit 702 may also constitute machine-readable storage media.

[0099] Although computer-readable storage medium 724 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" is intended to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of executable instructions. The term "computer-readable storage medium" is also intended to include any tangible medium capable of storing or encoding a set of instructions for execution by a computer that causes the computer to perform any one or more of the methods described herein. The term "computer-readable storage medium" is intended to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0100] The methods, components, and features described herein may be implemented by individual hardware components or may be integrated into the functions of other hardware components, such as ASICs, FPGAs, DSPs, or similar devices. In addition, the methods, components, and features may be implemented by firmware modules or functional circuits within a hardware device. Furthermore, the methods, components, and features may be implemented in any combination of hardware devices and computer program components, or in a computer program.

[0101] Unless otherwise indicated, terms such as "receive," "perform," "provide," "obtain," "cause," "access," "determine," "add," "use," "train," and the like refer to actions and processes performed or implemented by a computer system that manipulate data represented as physical (electronic) quantities in the registers and memory of the computer system and transform such data into other data similarly represented as physical quantities in the memory or registers of the computer system or other such information storage, transmission or display devices. Also, terms such as "first," "second," "third," "fourth," and the like, as used herein, are intended as labels to distinguish different elements and may not have a sequential meaning according to their numerical designations.

[0102] The examples described herein also relate to an apparatus for carrying out the methods described herein. The apparatus may be specially constructed to carry out the methods described herein, or may comprise a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0103] The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform the methods described herein and / or each of their individual functions, routines, subroutines, or operations. Examples of the structure of a variety of these systems are set forth in the description above.

[0104] The above description is intended to be illustrative, and not limiting. Although the present disclosure has been described with reference to certain illustrative examples and embodiments, it will be recognized that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. initializing, by a processor, an agent of a prediction subsystem of a substrate manufacturing system to select an action to perform in a simulated environment associated with the substrate manufacturing system; initiating a simulation of the selected action in the simulation environment; in response to pausing the simulation, obtaining output data based on an environmental state associated with the simulation; updating the agent so as to be configured to generate one or more dispatching decisions indicative of a time to commence processing of one or more substrates in the substrate manufacturing system based on the output data; A method comprising:

2. receiving a request to initiate a set of operations to be performed on a set of candidate substrates in the substrate manufacturing system, the set of operations including one or more operations each having one or more time constraints; obtaining current data relating to a current state of the substrate manufacturing system; providing said current data as input to said agent to obtain one or more outputs indicative of a time to process said candidate set of substrates; commencing said sequence of operations on said candidate substrate set at said determined time; The method of claim 1 further comprising:

3. receiving a request to initiate a set of operations to be performed on a set of candidate substrates in the substrate manufacturing system, the set of operations including one or more operations each having one or more time constraints; obtaining current data relating to a current state of the substrate manufacturing system; providing said current data as input to said agent to obtain one or more outputs indicative of a subset of substrates to be processed from a set of candidate substrates; initiating the sequence of operations on a subset of the substrates; The method of claim 1 further comprising:

4. The method of claim 1 , wherein the agent comprises a deep reinforcement learning model.

5. selecting a new action based on the output data; initiating the simulation of the new action in the simulation environment; The method of claim 1 further comprising:

6. The method of claim 1 , wherein the output data comprises environmental state data and compensation data, and the environmental state data comprises at least one of manufacturing equipment characteristics, manufacturing equipment observations, latency observations, or performance observations.

7. The method of claim 1 , wherein the action comprises a decision to at least one of: start processing of one or more substrates, not start processing of the one or more substrates, or start processing of a subset of the one or more substrates.

8. 1. An electronic device manufacturing system, comprising: A memory device; operatively coupled to the memory device; initializing an agent of a prediction subsystem of the manufacturing system to select an action to perform in a simulation environment associated with the manufacturing system; initiating a simulation of the selected action in the simulation environment; in response to pausing the simulation, obtaining output data based on an environmental state associated with the simulation; and updating the agent so as to be configured to generate one or more dispatching decisions indicative of a time to commence processing of one or more substrates in the manufacturing system based on the output data; A processing unit for performing operations including: An electronic device manufacturing system comprising:

9. The operation is receiving a request to initiate a set of operations to be performed on a set of candidate substrates in the manufacturing system, the set of operations including one or more operations each having one or more time constraints; obtaining current data regarding a current state of the manufacturing system; providing said current data as input to said agent to obtain one or more outputs indicative of a time to process said set of candidate substrates; commencing said sequence of operations on said candidate substrate set at said determined time; The electronic device manufacturing system of claim 8 , further comprising:

10. The operation is receiving a request to initiate a set of operations to be performed on a set of candidate substrates in the manufacturing system, the set of operations including one or more operations each having one or more time constraints; obtaining current data regarding a current state of the manufacturing system; providing said current data as input to said agent to obtain one or more outputs indicative of a subset of substrates to be processed from a set of candidate substrates; initiating the sequence of operations on a subset of the substrates; The electronic device manufacturing system of claim 8 , further comprising:

11. The electronic device manufacturing system of claim 8 , wherein the agent comprises a deep reinforcement learning model.

12. The operation is selecting a new action based on the output data; initiating the simulation of the new action in the simulation environment; The electronic device manufacturing system of claim 8 , further comprising:

13. 9. The electronic device manufacturing system of claim 8, wherein the output data includes environmental state data and compensation data, and the environmental state data includes at least one of manufacturing equipment characteristics, manufacturing equipment observations, latency observations, or performance observations.

14. 10. The electronic device manufacturing system of claim 8, wherein the action comprises a decision to at least one of: start processing of one or more substrates, not start processing of the one or more substrates, or start processing of a subset of the one or more substrates.

15. receiving a request to initiate a set of operations to be performed on a set of candidate substrates in a substrate manufacturing system, the set of operations including one or more operations each having one or more time constraints; obtaining current data relating to a current state of the substrate manufacturing system; providing said current data as input to an agent to obtain one or more outputs indicative of a time to process said set of candidate substrates; initiating said sequence of operations on at least one of said set of candidate substrates, or on a subset of substrates, at said determined time; A method comprising:

16. training the agent, initializing the agent to select actions to perform in a simulated environment associated with the substrate manufacturing system; initiating a simulation of the selected action in the simulation environment; in response to pausing the simulation, obtaining output data based on an environmental state associated with the simulation; updating the agent so as to be configured to generate one or more dispatching decisions indicative of a time to commence processing of one or more substrates in the substrate manufacturing system based on the output data; 16. The method of claim 15, comprising:

17. The method of claim 15 , wherein the agent comprises a deep reinforcement learning model.

18. The method of claim 15 , wherein the output data includes environmental state data and reward data.

19. 20. The method of claim 18, wherein the environmental condition data includes at least one of manufacturing equipment characteristics, manufacturing equipment observations, latency observations, or capacity observations.

20. 16. The method of claim 15, wherein the action comprises a decision to at least one of: start processing of one or more substrates, not start processing of the one or more substrates, or start processing of a subset of the one or more substrates.

Images