Method and system for scheduling order of operating multiple machines at manufacturing facility

Abstract

Methods and systems are provided for scheduling an order of operating a plurality of machines at a manufacturing facility. The method includes operating a machine learning agent using a learning policy; assigning a risk level for each intervention event of the intervention policy during the above operation of the machine learning agent using the learning policy; for each state-action pair, performing offline learning of a state-action value function, the state-action value function defining an intervention risk of an intervention strategy; generating an integrated policy by combining the learning policy with the state-action value function; operating the machine learning agent using the synthetic policy; updating the state-action value function and the learning policy based on intervention of the intervention policy; and outputting the learning strategy after the updating.

Description

Technical Field

[0001] The present disclosure relates to reinforcement learning techniques. Background Art

[0002] Modern manufacturing facilities often produce a wide variety of products (e.g., batches of tens of thousands of products), each of which undergoes numerous processes across multiple machines. Consequently, jobs compete for resources such as machines and masks. Efficiently coordinating these machines to achieve high productivity and utilization is a challenge.

[0003] Some related art scheduling methods utilize one or more human operators. However, human-based scheduling requires extensive domain expertise and years of training to generate reasonable schedules.

[0004] Some related art scheduling methods utilize scheduling techniques based on reinforcement learning (RL). Reinforcement learning is a machine learning technique in which an artificial intelligence-based agent takes actions in a dynamic environment through trial and error to maximize collective rewards based on feedback generated based on the actions taken by the agent. Factories typically use a set of evaluation criteria to evaluate the scheduling quality given by a specific scheduling strategy, where the quality is determined by calculating the weighted sum of these criteria. However, it is not always feasible to enumerate all safety considerations during modeling, and when the deployed strategy may lead to harmful results, experts or schedulers may intervene during production (that is, intervention strategies and / or manual intervention may be forced to intervene to prioritize safety and ensure that the selected actions are safe). That is, when the agent explores and tries actions that may have negative consequences, the use of related art RL-based scheduling techniques may lead to unsafe or dangerous behavior, especially when the consequences of the actions are not well understood or when exploration is expensive or dangerous. If applied independently, the intervention strategy / manual intervention may reduce the performance of the optimal scheduling strategy.

[0005] Furthermore, some related art scheduling methods may utilize constrained Markov decision processes (CMDPs), but these methods do not guarantee safety during the agent's training phase, employ iterative methods that may be difficult to scale effectively, may encounter convergence issues during training, and require the smoothness and ergodicity of CMDPs. Furthermore, some related art scheduling methods utilize domain-specific heuristic rules, but the agent cannot inherently learn to be safe and avoid violating these heuristic rules during execution.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not constitute prior art. Summary of the Invention

[0007] The present disclosure relates to various methods. In one embodiment, the method includes: operating a machine learning agent using a learning strategy; during the operation of the machine learning agent using the learning strategy, assigning a risk level to each intervention event of the intervention strategy; for each state-action pair, performing offline learning of a state-action value function, the state-action value function defining the intervention risk of the intervention strategy; for each round, iteratively performing the following steps until convergence: generating a comprehensive strategy by combining the learning strategy with the state-action value function; operating the machine learning agent using the comprehensive strategy; and updating the state-action value function and the learning strategy based on the intervention of the intervention strategy; and outputting the learning strategy after the update.

[0008] Operating a machine learning agent using a learning strategy may be performed in a manufacturing facility comprising a plurality of machines and the learning strategy may determine an order in which to operate the plurality of machines, and the method may further include scheduling the above-mentioned order of operating the plurality of machines at the manufacturing facility by the machine learning agent operating using the learning strategy.

[0009] The method may also include scheduling jobs at a manufacturing facility having multiple machines and multiple masks that produces multiple product types based on the learned policy.

[0010] For each of the plurality of product types, the state of each state-action pair may be a job queue length, a waiting time of a first job, or an urgency of each of the jobs.

[0011] For each mask in the plurality of masks, the state of each state-action pair may be the number of available masks.

[0012] Operating a machine learning agent using a comprehensive policy can include collecting tuples (state, action, reward, next state) and tuples (state, action, risk level, next state).

[0013] The intervention policy may include rules for processing jobs with the longest waiting queues first.

[0014] The intervention policy may include a rule for first processing jobs that have a wait time exceeding a threshold wait time.

[0015] The intervention policy may include rules for processing jobs in a first-in, first-out (FIFO) manner.

[0016] The method may also include scheduling jobs at the manufacturing facility based on the learned policy.

[0017] The risk level may be 1 for high risk or 0.1 for low risk.

[0018] The present disclosure also relates to various embodiments of a method for scheduling jobs at a manufacturing facility having multiple machines and multiple masks that produces multiple product types. In one embodiment, the method includes: operating a machine learning agent using a learning policy; during the operation of the machine learning agent using the learning policy, assigning a risk level to each intervention event of an intervention policy; for each state-action pair, performing offline learning of a state-action value function that defines the intervention risk of the intervention policy; for each round, iteratively performing the following steps until convergence: generating a composite policy by combining the learning policy with the state-action value function; operating the machine learning agent using the composite policy; and updating the state-action value function and the learning policy based on the intervention of the intervention policy; outputting the learning policy; and scheduling jobs of the manufacturing facility using the machine learning agent operated using the learning policy.

[0019] This summary is provided to introduce features and concepts of embodiments of the present disclosure that will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. One or more of the described features or tasks can be combined with one or more other described features or tasks to provide a viable method or system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The features and advantages of the embodiments of the present disclosure will be better understood by referring to the following detailed description when considered in conjunction with the accompanying drawings, which are not necessarily drawn to scale.

[0021] Figure 1 is a schematic diagram of an apparatus configured to schedule operations in a manufacturing facility for manufacturing products according to one embodiment of the present disclosure.

[0022] Figure 2 is a schematic diagram of a reinforcement learning system for managing scheduling of operations of a manufacturing facility according to one embodiment of the present disclosure.

[0023] Figure 3 is a flowchart illustrating tasks of a method for generating a reinforcement learning strategy using a comprehensive intervention strategy according to one embodiment of the present disclosure.

[0024] Figure 4 It is a schematic diagram depicting the manufacture of a group of products processed by different machines according to a predetermined process sequence.

[0025] Figure 5 is a flow chart illustrating the tasks of a method of scheduling jobs at a manufacturing facility that uses multiple machines to produce multiple product types according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure relates to various methods for training machine learning agents to develop policies for scheduling manufacturing processes that mitigate the risk of unintended consequences, unsafe behaviors, catastrophic failures, and / or harmful actions occurring during the training phase (i.e., as the machine learning agent explores its environment). The present disclosure also relates to the integration of intervention policies, which are a set of rules or actions taken to prevent the machine learning agent from taking certain actions that may result in unsafe and / or undesirable consequences without unduly degrading the performance of the machine learning agent (i.e., the integration of intervention policies is configured to promote safety while maintaining high performance of the machine learning agent). Although the present disclosure is described with respect to policies for scheduling manufacturing processes, the improvements described herein can be used in any system that uses policies to determine the order of operations (e.g., other robotic control applications, navigation systems, autonomous driving applications, and large language model training).

[0027] In the detailed description that follows, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood by those skilled in the art that the disclosed aspects can be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure the subject matter disclosed herein.

[0028] References throughout this specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in conjunction with that embodiment may be included in at least one embodiment disclosed herein. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases of similar meaning) throughout this specification may not necessarily all refer to the same embodiment. Furthermore, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" should not be construed as necessarily superior or advantageous over other embodiments.

[0029] In addition, depending on the context of the discussion in this article, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should also be noted that the various drawings (including component drawings) shown and discussed in this article are for illustrative purposes only and are not drawn to scale. For example, for clarity, the size of some of the elements may be exaggerated relative to other elements. In addition, if deemed appropriate, reference numerals have been repeated between the drawings to indicate corresponding elements and / or similar elements.

[0030] The terms used in this document are used only for the purpose of describing some example embodiments and are not intended to limit the claimed subject matter. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "comprise" and / or "include" indicate the presence of the described features, wholes, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or groups thereof.

[0031] It will be understood that when an element or layer is referred to as being "on," "connected to," or "coupled to" another element or layer, the element or layer can be directly on, directly connected to, or coupled to the other element or layer, or there can be intervening elements or layers. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers. Like reference numerals refer to like elements throughout. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0032] As used herein, the terms "first," "second," and the like are used as labels for the nouns that follow them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless expressly so defined. In addition, the same figure numeral may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such usage is merely for simplicity of illustration and ease of discussion; it does not imply that the construction or architectural details of such components or units are the same in all embodiments or that such commonly referenced parts / modules are the only way to implement some of the example embodiments disclosed herein.

[0033] Unless otherwise defined, all terms (including technical and scientific terms) used in this document have the same meaning as commonly understood by those skilled in the art to which this subject matter belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and unless expressly defined as such in this document, these terms will not be interpreted in an idealized or overly formal sense.

[0034] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with the module. For example, software may be embodied as a software package, code, and / or instruction set or instructions, and the term "hardware," as used in any embodiment described herein, may include, for example, components, hardwired circuits, programmable circuits, state machine circuits, and / or firmware that stores instructions executed by programmable circuits, either alone or in any combination. These modules may be collectively or individually embodied as circuits that form part of a larger system (e.g., but not limited to, integrated circuits (ICs), systems on chips (SoCs), components, etc.).

[0035] Figure 1 1 is a schematic diagram of an electronic device 100 configured to schedule the operating sequence of various machines (e.g., deposition stations) in a manufacturing facility for manufacturing one or more products (e.g., display devices). In the illustrated embodiment, the electronic device 100 includes a processor 101, a non-volatile memory device (also referred to as memory) 102 (e.g., flash memory, ferroelectric random access memory (F-RAM), magnetostrictive RAM (MRAM), FeFET memory, and / or resistive RAM (ReRAM) memory) connected to the processor 101, an input device 103 and an output device 104 connected to the processor 101 and / or the non-volatile memory device 102, a battery 105 connected to the processor 101 and the non-volatile memory device 102, a communication module 106, and an antenna module 107. The battery 105 can supply power to at least one component of the electronic device 100. The battery 105 can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In one or more embodiments, at least one of the above-described components may be omitted from the electronic device 100 , and / or one or more other components may be added to the electronic device 100 .

[0036] The input device 103 can receive commands or data from outside the electronic device 100 (e.g., a user) to be used by another component of the electronic device 100 (e.g., the processor 101). The input device 103 may include, for example, a microphone, a mouse, or a keyboard. The input device 103 is configured to accept or receive information about one or more parameters related to the machines and / or products being produced in the manufacturing facility (e.g., status information about the machines in the manufacturing facility or status information about each type of product produced at the manufacturing facility (e.g., the length of the job queue, the waiting time for the first job, the tightness of the waiting queue, and / or the urgency of each job type)).

[0037] The output device 104 is configured to output the scheduling order of machines for producing one or more products (e.g., machine 4 first, then machine 2, then machine 7, etc.). The output device 104 can be a screen such as an LED display device. In one or more embodiments, the input device 103 and the output device 104 can be combined or integrated into an input / output (I / O) device. The output device 104 can provide information visually to the outside of the electronic device 100 (e.g., a user). The output device 104 may include, for example, a display, a holographic device, or a projector and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. The output device 104 may include a touch circuit suitable for detecting a touch or a sensor circuit suitable for measuring the intensity of the force caused by the touch (e.g., a pressure sensor).

[0038] The program may be stored as software in the non-volatile memory device 102 and may include, for example, an operating system (OS), middleware, or an application. The processor 101 may execute the program to control at least one other component of the electronic device 100 coupled to the processor 101 (e.g., a hardware or software component), and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 101 may load commands or data received from another component (e.g., the input device 103) into the non-volatile memory device 102, may process the commands or data stored in the non-volatile memory device 102, and may store the resulting data in the non-volatile memory device 102. In one or more embodiments, the non-volatile memory device 102 stores artificial intelligence (e.g., a reinforcement learning algorithm) that, when executed by the processor 101, is configured to receive status information about machines and / or products being produced in the manufacturing facility from the input device 103, learn and set a scheduling sequence for the machines, and then output the scheduling sequence on the output device 104.

[0039] The term "processor" is used herein to include any combination of hardware, firmware, and software for processing data or digital signals. The hardware of the processor may include, for example, an application-specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device such as a field programmable gate array (FPGA). In a processor, as used herein, each function is performed by hardware configured (i.e., hard-wired) to perform the function or by more general hardware (e.g., CPU) configured to run instructions stored in a non-transitory storage medium. The processor may be manufactured on a single printed wiring board (PWB) or distributed on multiple interconnected PWBs. The processor may include other processors; for example, the processor may include two processors (FPGA and CPU) interconnected on a PWB. The processor 101 may include a main processor (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently of the main processor or in conjunction with the main processor. Additionally or alternatively, the auxiliary processor may be adapted to consume less power than the main processor or to perform a specific function. The auxiliary processor may be implemented as a part of the main processor or as a part of the main processor. The auxiliary processor may replace the main processor when the main processor is in an inactive (e.g., sleep) state or control at least some of the functions or states related to at least one component (e.g., output device 104) among the components of the electronic device 100 together with the main processor when the main processor is in an active state (e.g., executing an application). The auxiliary processor (e.g., an image signal processor or a communication processor) may be implemented as a part of another component that is functionally related to the auxiliary processor.

[0040] The communication module 106 can be configured to support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 100 and an external electronic device (e.g., another electronic device or a server) and to perform communication via the established communication channel. The communication module 106 may include one or more communication processors that can operate independently of the processor 101 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. The communication module 106 may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a short-range communication network (e.g., The communication module 106 can communicate with an external electronic device or a server using a wireless fidelity (Wi-Fi) direct connection or an infrared data association (IrDA) standard) or a long-distance communication network (e.g., a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)). These different types of communication modules can be implemented as a single component (e.g., a single IC), or can be implemented as multiple components (e.g., multiple ICs) separated from each other. The communication module 106 can identify and authenticate the electronic device 100 in a communication network such as a short-range communication network or a long-distance communication network using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module.

[0041] The antenna module 107 can transmit signals or power to the outside of the electronic device 100 (e.g., an external electronic device), or can receive signals or power from the outside of the electronic device 100 (e.g., an external electronic device). The antenna module 107 may include one or more antennas, and thus, at least one antenna suitable for a communication scheme used in a communication network such as a long-range communication network or a short-range communication network can be selected, for example, by the communication module 106. Then, signals or power can be transmitted or received between the communication module 106 and the external electronic device via the selected at least one antenna.

[0042] Commands or data can be sent or received between the electronic device 100 and an external electronic device via a server coupled to a remote communication network. The external electronic device can be a device of the same or different type as the electronic device 100. All or some of the operations to be performed at the electronic device 100 can be performed at one or more of the external electronic devices. For example, if the electronic device 100 should perform a function or service automatically or in response to a request from a user or another device, then instead of performing the above-mentioned function or service or in addition to performing the above-mentioned function or service, the electronic device 100 can request one or more external electronic devices to perform at least a part of the above-mentioned function or service. The one or more external electronic devices that receive the request can perform the above-mentioned at least a part of the request in the above-mentioned function or service or additional functions or additional services related to the request, and can transmit the results of the execution to the electronic device 100. The electronic device 100 can provide a result (with or without further processing the result) as at least part of the reply to the request. To this end, for example, cloud computing, distributed computing or client-server computing technology can be used.

[0043] Figure 2 A reinforcement learning (RL) system 200 is shown for managing the scheduling of multiple hardware and / or software operations (e.g., operations for processing data or operations of machines in a manufacturing facility). For ease of explanation, the system is described with respect to the operations of machines in a manufacturing facility according to one embodiment of the present disclosure. Figure 2 , but the same approach can be applied to any system that determines the order of operations of multiple components. In the illustrated embodiment, the RL system 200 includes a machine learning agent (also referred to as an agent) 201 connected to an environment 202 (e.g., a machine in a manufacturing facility). The machine learning agent 201 can be stored in Figure 1 The non-volatile memory device 102 of the electronic device 100 is stored in the non-volatile memory device 102. The machine learning agent 201 is configured to take action a at time step t (for example, the machine learning agent 201 is configured to cause a machine in a manufacturing facility to take a certain action a). In response, the environment 202 is configured to transition from state S to state S', and after transitioning from state S to state S' under action a, the immediate reward R a (S, S') is generated. In one or more embodiments where environment 202 is a manufacturing facility that produces multiple products, for each type of product produced at the manufacturing facility, the status S may include the length of the job queue, the wait time for the first job, the tightness of the waiting queue, or the urgency of each job. In one or more embodiments, for each mask, the status S may include the number of available masks.

[0044] Reinforcement learning is modeled as a Markov decision process (MDP)<S,A,R,P,γ> , where S is a set of environment and agent states, A is a set of actions of the machine learning agent 201, P is the probability that the environment 202 will transition from state S to state S′ under action a in the set of actions A performed by the machine learning agent 201, and R a (S, S') is the immediate reward of the environment 202 after it transitions from state S to state S' under action a, and γ (where γ∈[0,1]) is a discount factor that determines whether the immediate reward (γ=0), the total reward (γ=1), or a balance of them (0<γ<1) is of interest. The state S of the environment 202 can be a safe state. or unsafe state In one or more embodiments, the machine learning agent 201 operates under a policy π that is configured to achieve a high cumulative reward (i.e., the sum of all rewards R that the machine learning agent 201 expects to receive when following the policy from the beginning to the end of the episode) with a minimum probability P of entering an unsafe state (i.e., max π V π (S0) makes (1-γ)∑ t＝0 P(S t ∈S safe |π)≥1-δ and Where δ is the threshold probability, V πis the reward function of policy π (i.e.,<S,A,R,P,γ> Reward MDP), and is the value-at-risk function of strategy π (i.e.,<S,A,C,P,γ> Cost MDP)). That is, V is the expected long-term reward from the current state S under policy π.

[0045] In one or more embodiments, RL system 200 is configured to achieve (e.g., guarantee) desired performance while also operating safely (e.g., reducing or minimizing the likelihood of intervention strategies). For example, in one or more embodiments, RL system 200 operates such that and in, is the strategy π * The reward value function of state S0 under π′ (S0) is the reward value function of state S0 under strategy π′, P(π * ) is in strategy π * The probability of intervention strategy intervention when the next operation is performed, is the risk value function of state S0 under the comprehensive strategy π′, is the risk value function of state S0 under the intervention policy μ, and ∈ is a greedy algorithm that defines the balance between exploration and exploitation. Asterisks (*) indicate the expected or optimal values ​​of the relevant parameters.

[0046] Figure 3 is a flowchart illustrating the tasks of a method 300 for generating a reinforcement learning policy using a comprehensive intervention strategy according to one embodiment of the present disclosure. In the illustrated embodiment, the method 300 includes, for each of a plurality of rounds (1...n), using a policy π θ (e.g., random policy) to unfold the trajectory of the machine learning agent until the intervention policy μ 1,...,K Intervention or maximum time limit is reached task 310. That is, during the deployment, state S can be a safe state or unsafe state If an unsafe state (S∈S unsafe ), then the intervention strategy μ will be executed 1，...,K , until a safe state (S∈S safe ). In one or more embodiments, the intervention strategy μ 1，...，K is a set of rules or actions that can be taken to intervene and prevent the machine learning agent from taking actions that may lead to unsafe or undesirable results. In this way, the intervention policy μ 1，...，K is configured to mitigate these risks and ensure that the actions taken by the machine learning agent comply with these predefined safety constraints.1，...,K A set of rules related to the production of products in a manufacturing facility may be included (e.g., processing jobs with the longest wait queue first, processing jobs with wait times exceeding a threshold wait time first, or processing jobs in a first-in, first-out (FIFO) manner).

[0047] In addition, in the illustrated embodiment, the method 300 further includes: for the intervention strategy μ 1,...，K For each intervention event, use the associated risk level r intervene To mark the intervention strategy μ 1,...，K Task 320 of the intervention signal (eg, task 320 includes recording state S, action a, risk r intervene , next state S'). That is, during the training of the machine learning agent, the intervention strategy μ 1，...，K Each intervention is marked or labeled with a corresponding risk level. The risk level is the risk level of each intervention strategy μ 1,...,K For example, in one or more embodiments, task 320 includes, in response to an intervention strategy μ indicating a high risk 1,...，K And use r intervene = 1 to flag an intervention signal or respond to an intervention strategy μ indicating low risk 1，...，K And use r intervene = 0.1 to mark the intervention signal.

[0048] In the illustrated embodiment, the method 300 further includes, for each state-action pair (S, a), performing offline learning to learn a state-action value function Q as a risk value table. safe (ie, quality function) task 330, the state-action value function Q safe Defining intervention strategies μ 1,...,K Given (S, a, r intervene , S′, a′), Q safe (S, a) = r + γQ safe (S′, a′), and given (S, a, r intervene , S′,Done),Q safe (S, a) = r + γmax a′ Q safe (S′, a′), where r is the intervention strategy μ 1，...，K The intervention risk of , and γ is a discount factor (where γ∈[0,1]). In offline learning, the machine learning agent learns from a pre-recorded dataset (i.e., policy training is separated from the data collection process), while in online learning, the machine learning agent interacts with the environment. In one or more embodiments, the risk level r intervenemay be sparse, and the method 300 may include using an exponential decay function or other inverse reinforcement learning (IRL) process to decompose the risk level r intervene In this manner, tasks 310, 320, and 330 are configured to enable secure data collection.

[0049] In the illustrated embodiment, the method 300 further includes initializing the strategy π θ and a task 340 of generating a comprehensive policy π′ configured to achieve both safety and performance, the comprehensive policy π′ being the policy π to be learned and the state-action value function Q safe (i.e., the risk value table learned in task 330). In one or more embodiments, the comprehensive strategy π' is defined as follows: Where τ is a set of security actions defined as follows: τ = {(S, a)∈S safe ×a:Q safe (S, a) < η}, where η is the learning rate.

[0050] In the illustrated embodiment, the method 300 further includes using the synthesis policy π′ to unfold the trajectory of the machine learning agent until the intervention policy μ 1，...，K The intervention or maximum time limit is reached task 360. Task 360 also includes collecting data as tuples (state S, action a, reward R, next state S') and tuples (state S, action a, risk level r, next state S').

[0051] Furthermore, in the illustrated embodiment, method 300 also includes an intervention strategy μ based on task 360. 1，...，K The intervention to update the state-action value function Q safe and the task of learning policy π370.

[0052] In one or more embodiments, method 300 includes repeatedly performing tasks 340 to 370 until convergence is achieved. After convergence, method 300 also includes generating or outputting a desired or optimal strategy. and the desired or optimal state-action value function The task 380 is performed. Asterisks (*) indicate expected or optimal values ​​of the relevant parameters. The learning policy π output in task 380 can then be used to schedule jobs or tasks in a manufacturing facility that uses multiple machines and processes to produce multiple products.

[0053] Figure 4The manufacturing of a set of products is depicted as being processed by different machines according to a learning policy π. Machines 1 through 12 may include different manufacturing machines operating according to the learning policy π. For example, a computing system may run a reinforcement learning model as described herein to determine the order in which machines 1 through 12 operate. Figure 4 The solid line in indicates that the order in which machines 1 to 12 are operated is determined based on the learning policy π. For example, the learning policy π output in task 380 may indicate a scheduling order 402 in which the job should be processed first by machine 1, then by machine 2, then by machine 5, then by machine 7, and finally by machine 12. In another example, the learning policy π output in task 380 may indicate a scheduling order 404 in which the job should be processed first by machine 3, then by machine 2, then by machine 4, then by machine 9, then by machine 5, and finally by machine 11. In another example, the learning policy π output in task 380 may indicate a scheduling order 406 in which the job should be processed first by machine 1, then by machine 6, then by machine 8, and finally by machine 10. The job scheduled using the learning policy π output in task 380 may be a deposition job that deposits a hard mask.

[0054] Figure 5 is to show that the scheduling uses multiple machines (e.g. Figure 4 , a flowchart of a method 500 for performing operations at a manufacturing facility that produces multiple product types using multiple deposition machines (including multiple masks). In the illustrated embodiment, the method 500 includes a task 510 of operating a machine learning agent using a learning strategy. The machine learning agent may be stored in a manufacturing facility. Figure 1 In the non-volatile memory device 102 of the electronic device 100.

[0055] In the illustrated embodiment, the method 500 further includes a task 520 of determining and outputting a desired or optimal learning strategy. The task 520 of determining and outputting a desired or optimal learning strategy may include the above referenced Figure 3 Tasks 310 through 380 are described in the method 300 shown in FIG.

[0056] In the illustrated embodiment, method 500 also includes the task 530 of scheduling jobs of the manufacturing facility using a machine learning agent operating using a learning strategy. Figure 4As shown in FIG5 , task 530 may include scheduling a job to be processed first by machine 1, then by machine 2, then by machine 5, then by machine 7, and finally by machine 12. In one or more embodiments, task 530 may include scheduling a job to be processed first by machine 3, then by machine 2, then by machine 4, then by machine 9, then by machine 5, and finally by machine 11. In one or more embodiments, task 530 may include scheduling a job to be processed first by machine 1, then by machine 6, then by machine 8, and finally by machine 10. The job scheduled in task 530 may be a deposition job utilizing a deposited hard mask or any other job performed during the manufacture of a display device or any other device. For example, the job may include a first job performed by a first deposition station utilizing a first hard mask and a second job performed by a second deposition station utilizing a second hard mask, and task 530 may include scheduling the order in which the product is deposited utilizing the first and second deposition stations. In the illustrated embodiment, method 500 also includes task 540 of manufacturing the product according to the schedule.

[0057] The subject matter and operational embodiments described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of these. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included in them. Furthermore, although a computer storage medium is not a propagated signal, it may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium may also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or may be included in them. Furthermore, the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0058] Although this specification may include many specific implementation details, the implementation details should not be interpreted as limiting the scope of any claimed subject matter, but rather should be interpreted as descriptions of features specific to a particular embodiment. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described as functioning in certain combinations and even initially claimed as such above, one or more features in the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a variant of a sub-combination or sub-combination.

[0059] Similarly, although operation has been described in a specific order in the accompanying drawings, this should not be understood as requiring to perform such operation in the specific order shown or in a continuous order or requiring to perform all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the above-described embodiments should not be understood as requiring all to carry out such separation in all embodiments, and should be understood that described program components and system can usually be integrated together in a single software product or packaged in a plurality of software products.

[0060] Thus, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0061] Those skilled in the art will recognize that the innovative concepts described herein can be modified and varied over a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but rather should be defined by the claims.

Claims

1. A method for scheduling a sequence of operations of a plurality of machines at a manufacturing facility, comprising: Use learned policies to operate machine learning agents; assigning a risk level to each intervention event of an intervention strategy during said operation of said machine learning agent using said learning strategy; For each state-action pair, performing offline learning of a state-action value function, wherein the state-action value function defines the intervention risk of the intervention strategy; For each of the multiple rounds, the following steps are iteratively performed until convergence: generating a comprehensive policy by combining the learned policy with the state-action-value function; operating the machine learning agent using the integrated strategy; as well as updating the state-action-value function and the learning policy based on the intervention of the intervention policy; and outputting the learning strategy after the updating.

2. The method according to claim 1, wherein The operation of the machine learning agent using the learning strategy is performed in the manufacturing facility including the plurality of machines, and the learning strategy determines the order in which to operate the plurality of machines, and Wherein, the method further includes scheduling the order in which the plurality of machines at the manufacturing facility are operated by the machine learning agent operating using the learning strategy.

3. The method according to claim 2, wherein: For each of a plurality of product types produced by the plurality of machines, a state of each state-action pair is selected from the group consisting of a job queue length, a wait time of a first job, and an urgency of each of the jobs.

4. The method according to claim 3, wherein: The plurality of machines includes a plurality of masks, and wherein, for each mask in the plurality of masks, the state of each state-action pair is the number of available masks.

5. The method according to claim 1, wherein The operation of the machine learning agent using the integrated strategy includes collecting tuples (state, action, reward, next state) and tuples (state, action, risk level, next state).

6. The method according to claim 1, wherein The intervention strategy includes a rule for processing the job with the longest waiting queue first.

7. The method according to claim 1, wherein The intervention policy includes a rule for first processing jobs having a wait time exceeding a threshold wait time.

8. The method according to claim 1, wherein The intervention policy includes rules for processing jobs in a first-in, first-out manner.

9. The method according to any one of claims 1 to 8, wherein: For high risk, the risk level is 1.

10. The method according to any one of claims 1 to 8, wherein For low risk, the risk level is 0.

1.

11. A system configured to schedule a sequence of operations for a plurality of machines at a manufacturing facility, the system comprising: one or more processors; a non-volatile memory device storing instructions that, when executed by the one or more processors, cause the system to: operating a machine learning agent using the learning strategy in the manufacturing facility; assigning a risk level to each intervention event of an intervention strategy during said operation of said machine learning agent using said learning strategy; For each state-action pair, performing offline learning of a state-action value function, wherein the state-action value function defines the intervention risk of the intervention strategy; For each of the multiple rounds, the following steps are iteratively performed until convergence: generating a comprehensive policy by combining the learned policy with the state-action-value function; operating the machine learning agent using the integrated strategy; as well as updating the state-action-value function and the learning policy based on the intervention of the intervention policy; as well as The learning policy is output after the updating, the learning policy determining the order in which to operate the plurality of machines.

12. The system according to claim 11, wherein: The instructions, when executed by the one or more processors, further cause the system to schedule the sequence of operating the plurality of machines at the manufacturing facility by the machine learning agent operating using the learning policy.

13. The system of claim 12, further comprising: An input device is configured to accept or receive information regarding one or more parameters related to the plurality of machines and / or products being produced in the manufacturing facility.

14. The system of claim 12, further comprising: An output device is configured to output the order of operating the plurality of machines.

15. The system according to claim 12, wherein: For each of a plurality of product types produced by the plurality of machines, a state of each state-action pair is selected from the group consisting of a job queue length, a wait time of a first job, and an urgency of each of the jobs.

16. The system according to claim 15, wherein: The plurality of machines includes a plurality of masks, and wherein, for each mask in the plurality of masks, the state of each state-action pair is the number of available masks.

17. The system of claim 12, wherein: The instructions, when executed by the one or more processors, further cause the system to collect a tuple (state, action, reward, next state) and a tuple (state, action, risk level, next state).

18. The system of claim 12, wherein: The intervention strategy includes a rule for processing the job with the longest waiting queue first.

19. The system of claim 12, wherein: The intervention policy includes a rule for first processing jobs having a wait time exceeding a threshold wait time.

20. The system of claim 12, wherein: The intervention policy includes rules for processing jobs in a first-in, first-out manner.

Images