Schedule creation program generation method, schedule creation program generation device, schedule creation device, recording medium, generation program, schedule creation program, substrate processing device, and substrate processing system

Abstract

This schedule creation program generation method includes a step of repeating an experience step including a timetable acquisition step, a placement step, and a reward determination step through reinforcement learning. In the placement step, a planning element given to each first substrate of a normal lot and a planning element given to each second substrate of a priority lot are sequentially placed in a timetable, and a reward to be provided to the timetable in which all the planning elements have been placed is determined in the reward determination step. The reward is provided when processing for at least one first substrate is completed before completion of processing for the priority lot. The reward is also provided when processing for the priority lot is completed before completion of processing for the normal lot.

Description

Schedule creation program generation method, schedule creation program generation device, schedule creation device, recording medium, generation program, schedule creation program, substrate processing apparatus, and substrate processing system

[0001] The present invention relates to a schedule creation program generating method, a schedule creation program generating device, a schedule creation device, a recording medium, a generating program, a schedule creation program, a substrate processing apparatus, and a substrate processing system.

[0002] A substrate processing apparatus is known that, when a priority lot occurs that needs to be processed urgently before other lots, immediately starts processing wafers from the priority lot while continuing liquid processing in parallel with wafers from a normal lot that have already been processed (see, for example, Patent Document 1).

[0003] The substrate processing apparatus disclosed in Patent Document 1, when a priority lot occurs, determines the number of processing units to be allocated to processing wafers in the priority lot by referring to an allocation number setting table and a priority setting table. The allocation number setting table stores a standard allocation number of processing units selected according to the number of wafers stored in a FOUP of the priority lot. The priority setting table stores a coefficient by which the standard allocation number selected in the allocation number setting table is multiplied, associated with four levels of priority: "low, medium, high, and highest."

[0004] JP 2011-82276 A

[0005] However, there are limits to the efficiency that can be achieved by determining the number of processing units to be allocated to processing wafers in a priority lot using a fixed setting and then scheduling the processing of wafers in the priority lot and wafers in a regular lot that are being processed ahead of the priority lot.

[0006] An object of the present invention is to provide a schedule creation program generation method, a schedule creation program generation device, a schedule creation device, a recording medium, a generation program, a schedule creation program, a substrate processing apparatus, and a substrate processing system that can create a time schedule for more efficiently processing lots that should be processed with priority and regular lots that are being processed in advance.

[0007] According to one aspect of the present invention, a schedule creation program generation method uses reinforcement learning to generate a schedule creation program for creating a time schedule for processing multiple substrates using a substrate processing apparatus. The substrate processing apparatus includes a container mounting unit, multiple substrate processing units, and a transport unit. The container mounting unit mounts multiple substrate housing containers, each housing one or more substrates constituting a single lot. Each of the multiple substrate processing units processes the substrate one by one. The transport unit transports the substrates between the multiple substrate housing containers and the multiple substrate processing units. The substrate processing apparatus completes processing for the lot by performing the substrate processing on the substrates constituting the lot one by one. The schedule creation program generates an interruption schedule in response to a situation in which a second substrate housing container containing a priority lot with a higher priority is placed on the container mounting unit when all or part of first substrates constituting a normal lot with a normal priority remain in a first substrate housing container previously placed on the container mounting unit. The interrupt schedule represents a time schedule for processing the first substrate remaining in the first substrate container and the multiple second substrates constituting the priority lot by the substrate processing apparatus. The schedule creation program generation method includes a process of increasing cumulative rewards by repeating an empirical step including a timetable acquisition process, an allocation process, and a reward determination process using the reinforcement learning. The timetable acquisition process represents a process of acquiring a timetable for defining the interrupt schedule. The allocation process represents a process of sequentially arranging multiple plan elements assigned to each of the first substrates remaining in the first substrate container and multiple plan elements assigned to each of the multiple second substrates in the timetable, thereby sequentially changing the state of the timetable. The reward determination process represents a process of determining a reward to be awarded for the final state of the timetable based on the final state of the timetable in which all the plan elements have been arranged and conditions for awarding a reward. The final state of the timetable corresponds to the interrupt schedule.The cumulative reward indicates the total value of the rewards awarded for the final state of the timetable. The conditions for awarding the reward include a first reward condition and a second reward condition. The first reward condition indicates that a first reward is awarded when the final state of the timetable indicates a state in which processing for at least one of the first substrates remaining in the first substrate housing container is completed before processing for the priority lot is completed. The second reward condition indicates that a second reward, the reward value of which is smaller than the first reward, is awarded when the final state of the timetable indicates a state in which processing for the priority lot is completed before processing for the regular lot is completed.

[0008] In one embodiment, the timetable acquisition step includes a step of acquiring a timetable in which plan elements assigned to each of the first substrates remaining in the substrate processing apparatus are arranged when the first substrates remain in the substrate processing apparatus, and the schedule creation program creates the interrupt schedule without changing the time schedule of the first substrates remaining in the substrate processing apparatus.

[0009] In one embodiment, when all of the first substrates that make up the regular lot remain in the first substrate storage container, the first compensation condition indicates a condition for granting the first compensation when the final state of the timetable indicates a state in which processing has been completed for a number of the first substrates that is set based on the number of the second substrates that make up the priority lot before processing for the priority lot is completed.

[0010] In one embodiment, the substrate processing apparatus is capable of performing the substrate processing on two or more of the substrates in parallel, and the first remuneration condition indicates that the first remuneration is to be awarded when the final state of the timetable indicates a state in which the substrate processing on the first substrate is being performed in parallel with the substrate processing on at least one of the second substrates.

[0011] In one embodiment, the conditions for awarding the reward further include a third reward condition, which indicates that the earlier the processing of the priority lot is completed, the greater the value of the third reward to be awarded. The maximum value of the third reward is smaller than the value of the first reward.

[0012] In one embodiment, the conditions for granting the reward further include a fourth reward condition. The fourth reward condition indicates that the earlier the processing of the regular lot is completed, the greater the value of the fourth reward will be. The maximum value of the fourth reward is smaller than the value of the first reward.

[0013] In one embodiment, the substrate processing apparatus is capable of performing the substrate processing on two or more of the substrates in parallel. The conditions for providing the remuneration further include a fifth remuneration condition. The fifth remuneration condition indicates that a fifth remuneration is to be provided when the final state of the timetable indicates a state in which the substrate processing on the first substrate is being performed in parallel with the substrate processing on at least one second substrate.

[0014] According to another aspect of the present invention, a schedule creation program generation device generates a schedule creation program for creating a time schedule for processing a plurality of substrates by a substrate processing apparatus using reinforcement learning. The schedule creation program generation device includes a storage unit and a processing unit. The storage unit stores a generation program that defines the schedule creation program generation method. The processing unit executes the generation program to generate the schedule creation program.

[0015] According to yet another aspect of the present invention, a schedule creation device creates a time schedule for processing a plurality of substrates using a substrate processing apparatus. The schedule creation device includes a storage unit and a processing unit. The storage unit stores a schedule creation program generated based on the schedule creation program generation method described above. The processing unit executes the schedule creation program to create the interrupt schedule.

[0016] According to yet another aspect of the present invention, a computer-readable recording medium stores a generation program that defines the schedule creation program generation method described above.

[0017] According to yet another aspect of the present invention, a computer-readable recording medium is provided, which records a schedule creation program generated based on the above-described schedule creation program generation method.

[0018] According to yet another aspect of the present invention, a generating program is executable by a computer, and defines the above-described method for generating a schedule creation program.

[0019] According to yet another aspect of the present invention, a schedule creation program executable by a computer is generated based on the above-described method for generating a schedule creation program.

[0020] According to yet another aspect of the present invention, a substrate processing apparatus completes processing of one or more substrates constituting a lot by performing substrate processing on the substrates one by one. The substrate processing apparatus includes a container mounting unit, multiple substrate processing units, a transport unit, a memory unit, and a processing unit. Multiple substrate accommodation containers that accommodate the substrates constituting the lot are mounted on the container mounting unit. Each of the multiple substrate processing units performs the substrate processing on the substrate one by one. The transport unit transports the substrates between the multiple substrate accommodation containers and the multiple substrate processing units. The memory unit stores a generation program that defines the schedule creation program generation method. The processing unit executes the generation program to generate a schedule creation program for creating a time schedule for processing multiple substrates using the reinforcement learning. When all or part of a first substrate constituting a regular lot with a normal priority remains in a first substrate storage container that has been placed prior to the container loading section, the processing unit executes the schedule creation program in response to a second substrate storage container containing a priority lot with a higher priority than the normal lot being placed on the container loading section, and creates the interrupt schedule for processing the first substrate remaining in the first substrate storage container and the multiple second substrates constituting the priority lot.

[0021] According to yet another aspect of the present invention, a substrate processing apparatus completes processing of one or more substrates constituting a lot by performing substrate processing on the substrates one by one. The substrate processing apparatus includes a container mounting unit, multiple substrate processing units, a transport unit, a memory unit, and a processing unit. The container mounting unit mounts multiple substrate accommodation containers that accommodate the substrates constituting the lot. Each of the multiple substrate processing units performs the substrate processing on the substrate one by one. The transport unit transports the substrates between the multiple substrate accommodation containers and the multiple substrate processing units. The memory unit stores a schedule creation program generated based on the schedule creation program generation method. When all or some of the first substrates constituting a regular lot with a normal priority remain in a first substrate accommodation container that has been placed prior to the container mounting unit, the processing unit executes the schedule creation program in response to a second substrate accommodation container accommodating a priority lot with a higher priority than the regular lot being placed on the container mounting unit, to create the interrupt schedule for processing the first substrates remaining in the first substrate accommodation container and the multiple second substrates constituting the priority lot.

[0022] According to yet another aspect of the present invention, a substrate processing system includes a substrate processing apparatus and the schedule creation program generation device described above. The substrate processing apparatus performs substrate processing on one or more substrates constituting a lot, thereby completing processing for the lot. The schedule creation program generation device further includes a transmission unit that transmits the schedule creation program to the substrate processing apparatus. The substrate processing apparatus includes a container mounting unit, multiple substrate processing units, a transport unit, a receiving unit, and a processing unit. Multiple substrate accommodation containers that accommodate the substrates constituting the lot are mounted on the container mounting unit. The multiple substrate processing units each perform the substrate processing on the substrate one by one. The transport unit transports the substrates between the multiple substrate accommodation containers and the multiple substrate processing units. The receiving unit receives the schedule creation program transmitted from the transmission unit of the schedule creation program generation device. When all or part of a first substrate constituting a regular lot with a normal priority remains in a first substrate storage container that has been placed prior to the container loading section, the processing unit executes the schedule creation program in response to a second substrate storage container containing a priority lot with a higher priority than the normal lot being placed on the container loading section, and creates the interrupt schedule for processing the first substrate remaining in the first substrate storage container and the multiple second substrates constituting the priority lot.

[0023] According to yet another aspect of the present invention, a substrate processing system includes a substrate processing apparatus and the schedule creation device described above. The substrate processing apparatus completes processing for one or more substrates constituting a lot by performing substrate processing on the substrates one by one. The schedule creation device further includes a transmitter that transmits the interrupt schedule to the substrate processing apparatus. The substrate processing apparatus includes a container mounting unit, multiple substrate processing units, a transport unit, and a receiver. Multiple substrate housing containers that accommodate the substrates constituting the lot are mounted on the container mounting unit. Each of the multiple substrate processing units performs the substrate processing on the substrate one by one. The transport unit transports the substrates between the multiple substrate housing containers and the multiple substrate processing units. The receiver receives the interrupt schedule transmitted from the transmitter of the schedule creation device.

[0024] According to at least one aspect of the present invention, a schedule creation program generation method, a schedule creation program generation device, a schedule creation device, a recording medium, a generation program, a schedule creation program, a substrate processing apparatus, and a substrate processing system, it is possible to create a time schedule that more efficiently processes lots that should be processed with priority and regular lots that are being processed in advance.

[0025] 4A is a diagram showing a schedule creation program generation system including a schedule creation program generation device according to a first embodiment of the present invention. FIG. 4B is a diagram showing a schedule creation program generation system including a schedule creation program generation device according to a first embodiment of the present invention. FIG. 4C is a plan view schematically showing an example of the configuration of a substrate processing apparatus to which a schedule creation program according to a first embodiment of the present invention is applied. FIG. 4D is a diagram showing an example of a processing procedure, processing time, and plan elements. FIG. 4E is a diagram showing an example of a timetable corresponding to the substrate processing apparatus of FIG. 2. FIG. 4F is a diagram showing a continuation of the example of the timetable shown in FIG. 4A. FIG. 4F is a diagram showing an example of a processing flow for creating an interrupt schedule. FIG. 4G is a block diagram showing a reinforcement learning system. FIG. 4H is a flowchart showing a schedule creation program generation method according to a first embodiment of the present invention. FIG. 4I is a flowchart showing a timetable creation process. FIG. 4J is a flowchart showing a process for randomly selecting one of actions and arranging plan elements in the timetable. FIG. 4J is a flowchart showing a process for predicting an action that will maximize a reward and arranging plan elements in the timetable. FIG. 4I is a diagram showing a substrate processing system including a substrate processing apparatus according to a first embodiment of the present invention. FIG. 4I is a diagram showing a process flow executed by a control unit included in the substrate processing apparatus according to the first embodiment of the present invention. FIG. 4I is a diagram showing a process flow for creating an interrupt schedule. FIG. 4J is a diagram showing a substrate processing system including a substrate processing apparatus according to a second embodiment of the present invention. FIG. 4J is a diagram showing a substrate processing system according to a third embodiment of the present invention. FIG. 4J is a diagram showing a substrate processing system according to a fourth embodiment of the present invention.

[0026] Hereinafter, with reference to the drawings (FIGS. 1A to 16), embodiments of the schedule creation program generation method, schedule creation program generation device, schedule creation device, recording medium, generation program, schedule creation program, substrate processing apparatus, and substrate processing system of the present invention will be described. However, the present invention is not limited to the following embodiments, and can be embodied in various aspects without departing from the gist of the present invention. Note that duplicated explanations may be omitted as appropriate. Furthermore, in the drawings, the same or equivalent parts are designated by the same reference symbols, and explanations will not be repeated.

[0027] The "substrate" in the embodiments of the present invention can be any of various substrates, such as a semiconductor wafer, a glass substrate for a photomask, a glass substrate for a liquid crystal display, a glass substrate for a plasma display, a substrate for an FED (Field Emission Display), a substrate for an optical disk, a substrate for a magnetic disk, and a substrate for a magneto-optical disk. Hereinafter, the embodiments of the present invention will be described mainly using as examples a schedule creation program generation method, a schedule creation program generation device, a schedule creation device, a recording medium, a generation program, a schedule creation program, a substrate processing apparatus, and a substrate processing system used in processing a disc-shaped semiconductor wafer, but the present invention can also be applied to processing the various substrates exemplified above. Furthermore, various substrate shapes can also be applied.

[0028] [First Embodiment] Hereinafter, a first embodiment of the present invention will be described with reference to Figures 1A to 13. Figures 1A and 1B are diagrams showing a schedule creation program generation system 100A including a schedule creation program generation device 100 of this embodiment. In detail, Figure 1A shows the schedule creation program generation system 100A before a schedule creation program CP is created. Figure 1B shows the schedule creation program generation system 100A after the schedule creation program CP has been created.

[0029] As shown in FIG. 1A, a schedule creation program generation system 100A includes a schedule creation program generation device 100 and a recording medium 110.

[0030] The recording medium 110 is readable by a computer. A program (computer program) to be executed by a computer is non-temporarily recorded on the recording medium 110. The recording medium 110 stores a generating program GP. The generating program GP is a computer program that can be executed by a computer.

[0031] The recording medium 110 may be, for example, a medium including a semiconductor memory such as a secure digital (SD) memory card or a universal serial bus (USB) memory, or a medium including a magnetic disk such as a hard disk drive. Alternatively, the recording medium 110 may be an optical disk such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk. Alternatively, the recording medium 110 may be a main storage device or an auxiliary storage device installed in another computer system.

[0032] The schedule creation program generation device 100 generates a schedule creation program CP for creating a time schedule for processing multiple substrates W by a substrate processing apparatus WP based on the generation program GP. More specifically, the generation program GP includes a program for reinforcement learning. The schedule creation program generation device 100 generates the schedule creation program CP by reinforcement learning. The schedule creation program generation device 100 may be, for example, a general-purpose computer system or a dedicated computer system.

[0033] In detail, the schedule creation program CP is a computer program that creates a time schedule for processing by the substrate processing apparatus WP unprocessed substrates W of the normal lot and substrates W of the priority lot remaining in the substrate processing container CA that has docked prior to the substrate processing apparatus WP when a substrate processing container CA that contains a normal lot, which has a normal priority, is docked prior to the substrate processing apparatus WP and a substrate processing container CA that contains a priority lot, which has a higher priority than normal, is docked to the substrate processing apparatus WP, causing an interruption in processing of the priority lot.

[0034] The unprocessed substrates W of a normal lot remaining in a substrate accommodation container CA that is docked in advance to the substrate processing apparatus WP refer to the substrates W of the normal lot before being loaded into the substrate processing apparatus WP. The unprocessed substrates W refer to the substrates W before being subjected to substrate processing by the substrate processing unit PU, which will be described later with reference to Fig. 2. The priority lot refers to a lot that needs to be processed with priority over the normal lot.

[0035] Hereinafter, the time schedule for processing unprocessed substrates W from the regular lot and substrates W from the priority lot remaining in the substrate storage container CA that is docked prior to the substrate processing device WP by the substrate processing device WP may be referred to as the "interrupt schedule."

[0036] Furthermore, a substrate W constituting a normal lot may be referred to as a "normal substrate W1." Similarly, a substrate W constituting a priority lot may be referred to as a "priority substrate W2." Furthermore, an unprocessed normal substrate W1 may be referred to as an "unprocessed normal substrate W1b" or a "normal substrate W1b." Similarly, an unprocessed priority substrate W2 may be referred to as an "unprocessed priority substrate W2b" or a "priority substrate W2b." Furthermore, an unprocessed normal substrate W1b remaining in a substrate storage container CA may be referred to as a "container-remaining substrate W1b."

[0037] In this embodiment, the generation program GP includes a first generation program GP1 and a second generation program GP2. The schedule creation program generation device 100 generates a first schedule creation program CP1 through reinforcement learning based on the first generation program GP1, and generates a second schedule creation program CP2 through reinforcement learning based on the second generation program GP2. The first schedule creation program CP1 and the second schedule creation program CP2 are both computer programs that create interrupt schedules.

[0038] As shown in FIG. 1A, the schedule creation program generation device 100 includes an input unit 101, a storage unit 102, an interface unit 103, a display unit 104, and a processing unit 105.

[0039] The interface unit 103 exchanges information, data, or signals with the recording medium 110. Specifically, the interface unit 103 reads the generation programs GP (first generation program GP1 and second generation program GP2) from the recording medium 110 and inputs them to the calculation processing unit 105. As a result, the generation programs GP (first generation program GP1 and second generation program GP2) are installed in the schedule creation program generation device 100. Furthermore, as shown in FIG. 1B , the interface unit 103 causes the recording medium 110 to carry the schedule creation programs CP (first schedule creation program CP1 and second schedule creation program CP2).

[0040] For example, the interface unit 103 may be electrically connected to the recording medium 110 to input and output information, data, or signals to and from the recording medium 110. For example, the interface unit 103 may include a slot and a USB terminal. A card-shaped information carrier, such as an SD memory card, may be inserted into the slot. For example, a USB memory may be inserted into the USB terminal, or the other end of a USB cable, one end of which is electrically connected to a hard disk drive, may be inserted into the USB terminal. Alternatively, the interface unit 103 may include an optical disk drive. The optical disk drive reads information (data) from a CD (compact disk), a DVD, and / or a Blu-ray disk. The optical disk drive also writes information (data) to a CD, a DVD, and / or a Blu-ray disk.

[0041] The interface unit 103 may receive the generation program GP from another computer system. The interface unit 103 may also transmit the schedule creation program CP to another computer system. For example, the interface unit 103 may be communicatively connected to the other computer system via a cable, or may be communicatively connected to the other computer system via a line network such as the Internet.

[0042] The input unit 101 includes a user interface device operated by an operator. The input unit 101 inputs a signal corresponding to an operation by the operator to the arithmetic processing unit 105. The input unit 101 includes, for example, a keyboard and a mouse. The input unit 101 may include a touch sensor superimposed on the display surface of the display unit 104. A graphical user interface may be configured by superimposing the touch sensor on the display surface of the display unit 104.

[0043] For example, the operator can operate the input unit 101 to instruct the installation of the generation program GP. The operator can operate the input unit 101 to instruct the start of reinforcement learning. The operator can operate the input unit 101 to set a condition for terminating reinforcement learning. For example, a threshold value for the number of times that an interrupt schedule (time schedule) is repeatedly created (tried) may be set as a condition for terminating reinforcement learning. In this case, reinforcement learning ends when the number of times that an interrupt schedule is created (number of trials) reaches the set threshold value. Hereinafter, the threshold value set as a condition for terminating reinforcement learning may be referred to as a "first threshold value."

[0044] In this embodiment, one interruption schedule (time schedule) is created in one reinforcement learning step. In other words, the creation of one interruption schedule completes one reinforcement learning episode. Therefore, the number of times the creation (trial) of the interruption schedule is repeated corresponds to the number of times the reinforcement learning step is repeated.

[0045] The display unit 104 is controlled by the arithmetic processing unit 105 to display various screens. Specifically, the display unit 104 may display a learning curve. The learning curve indicates the relationship between the number of reinforcement learning steps and the cumulative reward. The display unit 104 includes a display device such as a liquid crystal display device or an organic electroluminescence (EL) display device. Note that the display unit 104 may be omitted.

[0046] The storage unit 102 has a main storage device. The main storage device includes, for example, a semiconductor memory. The storage unit 102 may further include an auxiliary storage device. The auxiliary storage device includes, for example, at least one of a semiconductor memory and a hard disk drive. The storage unit 102 may include removable media. The storage unit 102 stores various computer programs and various data. Specifically, the storage unit 102 stores a generation program GP. The storage unit 102 also stores a condition for terminating reinforcement learning. For example, the storage unit 102 stores a first threshold set by an operator operating the input unit 101 as a condition for terminating reinforcement learning.

[0047] As already described, the generation program GP includes a program for reinforcement learning. Specifically, the first generation program GP1 and the second generation program GP2 each include a program for reinforcement learning. The reinforcement learning algorithm is not particularly limited, but may be, for example, an algorithm conforming to Q-learning, the SARSA method, the policy gradient method, the Actor-Critic method, or the Monte Carlo method.

[0048] The neural network for reinforcement learning may include a neural network that performs deep learning. Specifically, the neural network may include a deep neural network (DNN), a deep Q-network (DQN), a transformer, a recurrent neural network (RNN), a convolutional neural network (CNN), or a quantum neural network (QNN). For example, a deep neural network includes an input layer, multiple intermediate layers (hidden layers), and an output layer.

[0049] The arithmetic processing unit 105 includes a processor. The arithmetic processing unit 105 may include, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a neural network processing unit (NPU), or a quantum computer. Alternatively, the arithmetic processing unit 105 may include a general-purpose arithmetic device or a dedicated arithmetic device. For example, the arithmetic processing unit 105 may include a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0050] The calculation processing unit 105 executes the generation program GP stored in the storage unit 102 to generate the schedule creation program CP. Specifically, the calculation processing unit 105 executes the first generation program GP1 stored in the storage unit 102 to generate the first schedule creation program CP1, and executes the second generation program GP2 stored in the storage unit 102 to generate the second schedule creation program CP2. The calculation processing unit 105 records the schedule creation programs CP (the first schedule creation program CP1 and the second schedule creation program CP2) on the recording medium 110.

[0051] Specifically, the arithmetic processing unit 105 creates a time schedule for processing a plurality of substrates W by the substrate processing apparatus WP by arranging a plurality of plan elements BL, which will be described with reference to Figure 3, in a time table TB, which will be described with reference to Figures 4A and 4B, in each step of reinforcement learning. More specifically, the arithmetic processing unit 105 repeatedly attempts to create an interrupt schedule by reinforcement learning. The plurality of substrates W include unprocessed normal substrates W1b (container remaining substrates W1b) remaining in the substrate accommodation container CA and unprocessed priority substrates W2b.

[0052] For example, when the calculation processing unit 105 repeatedly attempts to create a time schedule for processing 25 substrates W using the substrate processing device WP under the condition that four planning elements BL are placed in the time table TB for each substrate W, it places 100 planning elements BL in the time table TB at each step of reinforcement learning.

[0053] Furthermore, the calculation processing unit 105 refers to the final state of the timetable TB and the conditions for granting rewards, and acquires a cumulative reward each time it repeats an attempt to create an interrupt schedule. The conditions for granting rewards include a first reward condition and a second reward condition. The final state of the timetable TB indicates the timetable TB in which all plan elements BL are arranged. Here, all plan elements BL include multiple plan elements BL assigned to each of the container remaining substrates W1b and multiple plan elements BL assigned to each of the unprocessed priority substrates W2b. In other words, the final state of the timetable TB corresponds to the interrupt schedule. The cumulative reward indicates the total value of rewards assigned for the final state of the timetable TB.

[0054] The first remuneration condition indicates that the first remuneration is to be granted when the final state of the timetable TB indicates that processing of at least one of the container remaining substrates W1b is completed before processing of the priority lot is completed. The first remuneration is the remuneration with the largest remuneration value. By including the first remuneration condition in the conditions for granting the remuneration, it is possible to avoid stopping processing of the container remaining substrates W1b until processing of the priority lot is completed.

[0055] The completion of processing for a substrate W indicates a state in which the substrate W has been processed and has been loaded into the substrate accommodation container CA, and the completion of processing for a lot indicates a state in which substrate processing has been performed on all of the substrates W constituting the lot, and all of the substrates W (processed substrates W) constituting the lot have been loaded into the substrate accommodation container CA.

[0056] The second remuneration condition indicates that a second remuneration is to be granted when the final state of the timetable TB indicates that processing for a priority lot is completed before processing for a normal lot is completed. The value of the second remuneration is smaller than the first remuneration. The second remuneration may be the remuneration with the second largest remuneration value. By including the second remuneration condition in the conditions for granting a remuneration, it is possible to create a time schedule (interrupt schedule) that prioritizes processing of priority lots.

[0057] In this embodiment, the conditions for awarding a reward further include a third reward condition and a fourth reward condition. The third reward condition indicates that the earlier the time at which processing of a priority lot is completed, the larger the reward value of the third reward is awarded. The fourth reward condition indicates that the earlier the time at which processing of a normal lot is completed, the larger the reward value of the fourth reward is awarded. The maximum value of the third reward is smaller than the second reward. The maximum value of the fourth reward is smaller than the third reward.

[0058] By including the third reward condition in the conditions for granting rewards, it is possible to create a time schedule (interrupt schedule) in which processing for priority lots is completed in a shorter time. Furthermore, by including the fourth reward condition in the conditions for granting rewards, it is possible to create a time schedule (interrupt schedule) in which processing for normal lots is completed in a shorter time. Note that the third reward condition and the fourth reward condition may be specified by a reward function. The reward function indicates a function in which the reward value increases the earlier the time when processing for a lot is completed.

[0059] The arithmetic processing unit 105 adjusts the parameters (weighting coefficients) of the neural network so as to maximize the cumulative reward during the process of repeating reinforcement learning steps (trials). In other words, the arithmetic processing unit 105 adjusts the parameters of the neural network so that the priority lot is processed prior to the normal lot without stopping the processing of the container-remaining substrates W1b, and so that the processing of the priority lot and the normal lot is completed in a shorter time. For example, the parameters of the neural network are adjusted until the number of trials reaches a first threshold. As a result, a schedule creation program CP (trained model) is generated.

[0060] Specifically, the calculation processing unit 105 adjusts the parameters (weighting coefficients) of the neural network included in the first generation program GP1 to generate the first schedule creation program CP1 (trained model). Also, the calculation processing unit 105 adjusts the parameters (weighting coefficients) of the neural network included in the second generation program GP2 to generate the second schedule creation program CP2 (trained model).

[0061] By using the schedule creation programs CP (first schedule creation program CP1 and second schedule creation program CP2) created in this way, it is possible to create an interrupt schedule (time schedule) in which priority lots are processed in preference to normal lots without interrupting the processing of the container remaining substrates W1b, and in which the processing of the priority lots and normal lots is completed in a shorter time. Therefore, it is possible to create a time schedule in which the lots that should be processed with priority and the normal lots that are being processed in advance are processed more efficiently.

[0062] Next, an example of a substrate processing apparatus WP to which the schedule creation program CP of the present embodiment is applied will be described with reference to Fig. 2. Fig. 2 is a plan view schematically showing an example of the configuration of a substrate processing apparatus WP to which the schedule creation program CP of the present embodiment is applied.

[0063] The substrate processing apparatus WP is a single-wafer processing apparatus. The substrate processing apparatus WP performs substrate processing on one or more substrates W constituting one lot, one by one, to complete processing for one lot. The number of substrates W constituting one lot is, for example, 1 to 25. The substrate processing apparatus WP includes a container placement unit LPS, a plurality of substrate processing units PU, and a transport unit TR.

[0064] The container platform LPS is capable of holding a plurality of substrate accommodation containers CA. When a substrate accommodation container CA is placed on the container platform LPS, the substrate accommodation container CA is docked with the substrate processing apparatus WP.

[0065] The container mounting part LPS of the substrate processing apparatus WP shown in Fig. 2 includes four load ports LP (first load port LP1 to fourth load port LP4). A substrate accommodation container CA can be placed on each of the load ports LP. In the substrate processing apparatus WP shown in Fig. 2, four substrate accommodation containers CA can be placed on the container mounting part LPS. For example, the first substrate accommodation container CA1 to fourth substrate accommodation container CA4 may be placed on the container mounting part LPS.

[0066] The first substrate storage container CA1 stores normal substrates W1. Specifically, the first substrate storage container CA1 stores unprocessed normal substrates W1b. The second substrate storage container CA2 stores priority substrates W2. Specifically, the second substrate storage container CA2 stores unprocessed priority substrates W2b.

[0067] The third substrate accommodation container CA3 and the fourth substrate accommodation container CA4 are empty substrate accommodation containers CA. The third substrate accommodation container CA3 accommodates processed normal substrates W1. The fourth substrate accommodation container CA4 accommodates processed priority substrates W2. The processed substrates W refer to substrates W after substrate processing has been performed by the substrate processing unit PU.

[0068] For example, the first to fourth substrate accommodation containers CA1 to CA4 may be placed on the first to fourth load ports LP1 to LP4, respectively. Hereinafter, the processed normal substrate W1 may be referred to as the "processed normal substrate W1a." Similarly, the processed priority substrate W2 may be referred to as the "processed priority substrate W2a."

[0069] Each substrate storage container CA can store one or more substrates W that make up one lot. The substrate storage container CA may be, for example, a Front Opening Unified Pod (FOUP), a Standard Mechanical Interface (SMIF) pod, or an Open Cassette (OC).

[0070] Each substrate processing unit PU performs substrate processing on a single substrate W. The type of substrate processing is not particularly limited, and for example, the substrate processing unit PU may perform processing on the substrate W using a processing agent (processing liquid or processing gas, or processing liquid and processing gas), processing using electromagnetic waves such as ultraviolet rays, or physical cleaning processing (brush cleaning, spray nozzle cleaning, etc.). For example, the substrate processing unit PU may perform any of the following substrate processing on the substrate W: chemical cleaning processing, brush cleaning processing, wet etching processing, dry etching processing, photosensitive film coating processing, development processing, annealing processing, and drawing processing.

[0071] The transport part TR is capable of transporting substrates W between multiple substrate accommodation containers CA and multiple substrate processing units PU. In the example shown in Figure 2, the substrate processing apparatus WP includes 12 substrate processing units PU (first substrate processing unit PU1 to twelfth substrate processing unit PU12). Therefore, in the substrate processing apparatus WP shown in Figure 2, the transport part TR can transport substrates W between four substrate accommodation containers CA and 12 substrate processing units PU. The transport part TR includes, for example, an indexer robot IR, a transfer part PS, and a transport robot CR.

[0072] The indexer robot IR transports unprocessed substrates W from the substrate storage container CA to the passing part PS. The indexer robot IR also transports processed substrates W from the passing part PS to the substrate storage container CA. For example, the indexer robot IR transports an unprocessed normal substrate W1b from the first substrate storage container CA1 to the passing part PS, and an unprocessed priority substrate W2b from the second substrate storage container CA2 to the passing part PS. The indexer robot IR also transports a processed normal substrate W1a from the passing part PS to the third substrate storage container CA3, and transports a processed priority substrate W2a from the passing part PS to the fourth substrate storage container CA4.

[0073] Specifically, the indexer robot IR has two hands (hands 8A and 8B). Each of the hands 8A and 8B holds one substrate W. More specifically, the hand 8A holds an unprocessed substrate W. The hand 8B holds a processed substrate W.

[0074] The passing part PS has a plurality of shelves for supporting substrates W. Specifically, the passing part PS has at least one shelf for supporting unprocessed substrates W and at least one shelf for supporting processed substrates W. In this embodiment, the passing part PS has one shelf for supporting unprocessed substrates W and one shelf for supporting processed substrates W. Hereinafter, the shelf for supporting unprocessed substrates W may be referred to as "shelf PS1." Furthermore, the shelf for supporting processed substrates W may be referred to as "shelf PS2."

[0075] The transport robot CR transports an unprocessed substrate W from the passing part PS to one of the substrate processing parts PU. The transport robot CR also transports a processed substrate W from the substrate processing part PU to the passing part PS. Substrate processing for one substrate W is performed by one of the substrate processing parts PU.

[0076] Specifically, the transport robot CR has two hands (hands 13A and 13B). Each of the hands 13A and 13B holds one substrate W. More specifically, the hand 13A holds an unprocessed substrate W. The hand 13B holds a processed substrate W.

[0077] As shown in Fig. 2, the multiple substrate processing units PU form multiple towers TW that are arranged to surround the transport robot CR in a plan view. In the example shown in Fig. 2, the substrate processing apparatus WP has four towers TW (first tower TW1 to fourth tower TW4). Each tower TW includes multiple substrate processing units PU stacked one above the other. In the example shown in Fig. 2, each tower TW includes three substrate processing units PU stacked one above the other.

[0078] Specifically, the first substrate processing unit PU1 to the third substrate processing unit PU3 form the first tower TW1, the fourth substrate processing unit PU4 to the sixth substrate processing unit PU6 form the second tower TW2, the seventh substrate processing unit PU7 to the ninth substrate processing unit PU9 form the third tower TW3, and the tenth substrate processing unit PU10 to the twelfth substrate processing unit PU12 form the fourth tower TW4.

[0079] Next, the processing procedure PD, processing time PT, and plan element BL will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the processing procedure PD, processing time PT, and plan element BL. In detail, Fig. 3 shows the processing procedure PD, processing time PT, and plan element BL corresponding to the substrate processing apparatus WP of Fig. 2. Information indicating the processing procedure PD, information indicating the processing time PT, and information indicating the plan element BL are associated with each other and stored in the storage unit 102 described with reference to Figs. 1A and 1B.

[0080] The processing procedure PD indicates the procedure of processing performed by the substrate processing apparatus WP. As shown in Fig. 3, the processing procedure PD corresponding to the substrate processing apparatus WP in Fig. 2 includes processing pattern A to processing pattern M. Processing pattern A to processing pattern M are performed in this order on one substrate W. The processing procedure PD indicates the flow of processing (processing flow) performed on one substrate W. In a time table TB, which will be described with reference to Figs. 4A and 4B, processing pattern A to processing pattern M are arranged in this order along the time axis of the time table TB.

[0081] Processing pattern A shows a process in which the indexer robot IR takes out an unprocessed substrate W from a substrate storage container CA. Processing pattern B shows a process in which the indexer robot IR transports an unprocessed substrate W and loads the unprocessed substrate W into the transfer part PS. While processing pattern A and processing pattern B are being executed, the hand 8A of the indexer robot IR is occupied by one substrate W.

[0082] Processing pattern C indicates a process in which an unprocessed substrate W is carried into the transfer part PS. Processing pattern D indicates a process in which an unprocessed substrate W is carried out from the transfer part PS. While processing pattern C and processing pattern D are being executed, shelf PS1 of the transfer part PS is occupied by one substrate W.

[0083] Processing pattern E shows a process in which the transport robot CR takes out an unprocessed substrate W from the transfer part PS. Processing pattern F shows a process in which the transport robot CR transports an unprocessed substrate W and carries it into the substrate processing unit PU. While processing pattern E and processing pattern F are being executed, the hand 13A of the transport robot CR is occupied by one substrate W.

[0084] The processing pattern G indicates that the substrate processing unit PU performs substrate processing. While the processing pattern G is being performed, the substrate processing unit PU is occupied by one substrate W.

[0085] Processing pattern H indicates a process in which the transport robot CR unloads the processed substrate W from the substrate processing unit PU. Processing pattern I indicates a process in which the transport robot CR transports the processed substrate W and loads it into the transfer part PS. While processing pattern H and processing pattern I are being executed, the hand 13B of the transport robot CR is occupied by one substrate W.

[0086] Processing pattern J indicates a process in which a processed substrate W is carried into the passing part PS. Processing pattern K indicates a process in which a processed substrate W is carried out from the passing part PS. While processing pattern J and processing pattern K are being executed, shelf PS2 of the passing part PS is occupied by one substrate W.

[0087] Processing pattern L indicates a process in which the indexer robot IR takes out the processed substrate W from the transfer part PS. Processing pattern M indicates a process in which the indexer robot IR transports the processed substrate W and loads it into a substrate accommodation container CA. While processing pattern L and processing pattern M are being executed, the hand 8B of the indexer robot IR is occupied by one substrate W.

[0088] As explained above with reference to Fig. 3, one substrate W sequentially occupies a plurality of components included in the substrate processing apparatus WP. In this way, processing patterns A to M indicate occupancy information for one substrate W. In the substrate processing apparatus WP of Fig. 2, one substrate W sequentially occupies any one of the hands 8A and 8B of the indexer robot IR, the shelves PS1 and PS2 of the transfer part PS, the hands 13A and 13B of the transport robot CR, and the substrate processing unit PU.

[0089] Next, the processing time PT will be described. The processing time PT indicates the duration of each process performed by the substrate processing apparatus WP. More specifically, the processing time PT indicates the duration of a process performed by each component included in the substrate processing apparatus WP. In other words, the processing time PT indicates the duration over which each component included in the substrate processing apparatus WP is occupied by one substrate W.

[0090] 3, the processing time PT corresponding to the substrate processing apparatus WP in FIG. 2 includes processing times X1 to X13. Processing times X1 to X13 indicate the time lengths of processing patterns A to M, respectively. Processing times X1 to X13 are associated with processing patterns A to M, respectively, and stored in the storage unit 102 described with reference to FIGS. 1A and 1B. Of processing times X1 to X13, processing times X1 to X6 and processing times X8 to X13 indicate the time lengths required for transporting the substrate W. Processing time X7 indicates the time length required for substrate processing. The time lengths of processing times X1 to X6 and processing times X8 to X13 are shorter than processing time X7.

[0091] Next, the plan elements BL will be described. As shown in Fig. 3, the processing procedure PD is divided into a plurality of plan elements BL. Each plan element BL includes at least one of the processing patterns included in the processing procedure PD. In the example shown in Fig. 3, processing patterns A to M are divided into four plan elements BL (first plan element BL1 to fourth plan element BL4). Each plan element BL indicates a component that is continuously occupied by one substrate W. The plurality of plan elements BL are given in advance for the processing procedure PD. In other words, a plurality of plan elements BL are given in advance for one substrate W.

[0092] The first plan element BL1 includes processing patterns A to C. That is, the first plan element BL1 indicates a plan for transporting an unprocessed substrate W from a substrate storage container CA to a shelf PS1 of the transfer part PS by the indexer robot IR. The first plan element BL1 also indicates that one substrate W successively occupies the hand 8A of the indexer robot IR and the shelf PS1 of the transfer part PS.

[0093] The second plan element BL2 includes processing patterns D to G. In other words, the second plan element BL2 indicates a plan in which an unprocessed substrate W is transported by the transport robot CR from the shelf PS1 of the transfer unit PS to the substrate processing unit PU, and then processed by the substrate processing unit PU. The second plan element BL2 also indicates that one substrate W consecutively occupies the shelf PS1 of the transfer unit PS, the hand 13A of the transport robot CR, and any one of the plurality of substrate processing units PU.

[0094] The third plan element BL3 includes processing patterns H to J. That is, the third plan element BL3 indicates a plan for transporting a processed substrate W from the substrate processing unit PU to the shelf PS2 of the transfer unit PS by the transport robot CR. In other words, the third plan element BL3 indicates that one substrate W successively occupies the hand 13B of the transport robot CR and the shelf PS2 of the transfer unit PS.

[0095] The fourth plan element BL4 includes processing patterns K to M. That is, the fourth plan element BL4 indicates a plan for transporting a processed substrate W from the shelf PS2 of the transfer part PS to a substrate accommodation container CA. The fourth plan element BL4 also indicates that one substrate W successively occupies the shelf PS2 of the transfer part PS and the hand 8B of the indexer robot IR.

[0096] 1A and 1B arranges a plurality of plan elements BL in a time table TB in accordance with the order of the processing procedures PD. More specifically, the calculation processing unit 105 arranges process patterns A to M in the time table TB for each plan element BL in accordance with the order of the processing procedures PD. By arranging process patterns A to M in the time table TB for each plan element BL, physically impossible actions can be prohibited in the substrate processing apparatus WP.

[0097] Next, the constraints will be explained. Table 1 below shows an example of the constraints corresponding to the substrate processing apparatus WP in FIG. 2. The constraints indicate conditions for arranging the plan elements BL in the timetable TB. The constraints depend on the configuration of the substrate processing apparatus WP. For example, the constraints include a condition that prohibits physically impossible actions in the substrate processing apparatus WP.

[0098] 1A and 1B arranges a plurality of plan elements BL (processing pattern A to processing pattern M) in a timetable TB by referring to constraint conditions (constraint conditions 1 to 9). The constraint conditions are stored in the storage unit 102.

[0099] 1A to 3 and Table 1, when arranging multiple plan elements BL in the timetable TB, the calculation processing unit 105 refers to information indicating the processing procedure PD, information indicating the processing time PT, information indicating the plan elements BL, and constraints. Furthermore, when arranging multiple plan elements BL in the timetable TB, the calculation processing unit 105 refers to number information indicating the number of substrates W to be processed and information related to substrate processing. The information related to substrate processing indicates the substrate processing procedure to be performed on the substrates W to be processed and the processing conditions for that substrate processing.

[0100] Specifically, when arranging the plurality of plan elements BL in the timetable TB, the calculation processing unit 105 refers to first number information indicating the number of unprocessed normal substrates W1b (container remaining substrates W1b) present in the substrate accommodation container CA (first substrate accommodation container CA1) that accommodates the normal lot, and second number information indicating the number of unprocessed priority substrates W2b. In addition, the calculation processing unit 105 refers to information regarding the substrate processing to be performed on the normal substrates W1 and information regarding the substrate processing to be performed on the priority substrates W2.

[0101] Specifically, the information regarding the substrate processing to be performed on the normal substrate W1 indicates the procedure for the substrate processing to be performed on the normal substrate W1 and the processing conditions for that substrate processing. The information regarding the substrate processing to be performed on the priority substrate W2 indicates the procedure for the substrate processing to be performed on the priority substrate W2 and the processing conditions for that substrate processing. The procedure for the substrate processing to be performed on the normal substrate W1 may be the same as or different from the procedure for the substrate processing to be performed on the priority substrate W2. Furthermore, the processing conditions for the substrate processing to be performed on the normal substrate W1 may be the same as or different from the processing conditions for the substrate processing to be performed on the priority substrate W2.

[0102] The unprocessed normal substrates W1b (container-remaining substrates W1b) present in the first substrate accommodation container CA1 correspond to the unprocessed normal substrates W1b remaining in the first substrate accommodation container CA1 when the priority lot interruption occurs, among the normal substrates W1. Hereinafter, the first number information may be referred to as "container-remaining substrates W1b number information." Furthermore, the second number information may be referred to as "priority substrates W2 number information."

[0103] The calculation processing unit 105 refers to the information on the number of container residual substrates W1b and places a number of planning elements BL corresponding to the number of unprocessed normal substrates W1b (container residual substrates W1b) present in the first substrate storage container CA1 on the timetable TB, and refers to the information on the number of priority substrates W2 and places a number of planning elements BL corresponding to the number of unprocessed priority substrates W2b on the timetable TB.

[0104] Next, the timetable TB will be described with reference to FIGS. 1A to 4A and 4B. FIGS. 4A and 4B are diagrams showing an example of the timetable TB corresponding to the substrate processing apparatus WP of FIG. 2. FIG. 4A shows a portion of the timetable TB, and FIG. 4B shows the remainder of the timetable TB. As shown in FIGS. 4A and 4B, a plurality of plan elements BL are arranged in the timetable TB. Each of the plan elements BL is arranged in the timetable TB and exclusively occupies at least one of a plurality of components included in the substrate processing apparatus WP. By arranging the plurality of plan elements BL in the timetable TB, a time schedule for processing a plurality of substrates W by the substrate processing apparatus WP is defined.

[0105] Specifically, the horizontal axis of the timetable TB represents time. The timetable TB shown in FIGS. 4A and 4B represents time t1 to time t16. In the timetable TB, processing patterns corresponding to each component of the substrate processing apparatus WP are arranged along the time axis of the timetable TB. In the timetable TB shown in FIGS. 4A and 4B, the first planning element BL1 to the fourth planning element BL4 (processing pattern A to processing pattern M) described with reference to FIG. 3 are arranged based on constraints 1 to 9 shown in Table 1. The timetable TB is, for example, a Gantt chart. Note that processing pattern A to processing pattern M arranged in the timetable TB may be numerical information or image information.

[0106] In detail, the calculation processing unit 105 adds information to the first plan elements BL1 to the fourth plan elements BL4 (processing pattern A to processing pattern M) indicating whether the first plan elements BL1 to the fourth plan elements BL4 (processing pattern A to processing pattern M) are assigned to the container remaining substrate W1b or the priority substrate W2b, and then arranges the first plan elements BL1 to the fourth plan elements BL4 in the timetable TB.

[0107] 4A and 4B show an example of a timetable TB in which the first plan element BL1 to the fourth plan element BL4 assigned to the first priority substrate W2b and the first plan element BL1 to the fourth plan element BL4 assigned to the second priority substrate W2b are arranged. Specifically, in the example shown in FIGS. 4A and 4B, "W2" indicates that the priority substrate W2.

[0108] 4A and 4B, for ease of understanding, the order of the priority substrates W2b to be processed is indicated. Specifically, in the example shown in FIGS. 4A and 4B, "(1)" after "W2" indicates the first plan element BL1 to the fourth plan element BL4 assigned to the first priority substrate W2b, and "(2)" after "W2" indicates the first plan element BL1 to the fourth plan element BL4 assigned to the second priority substrate W2b.

[0109] As shown in Figures 4A and 4B, the substrate processing apparatus WP can perform substrate processing on two or more substrates W in parallel. Specifically, two or more substrate processing units PU can perform substrate processing in parallel. The example shown in Figures 4A and 4B shows a time schedule in which the second substrate processing unit PU2 and the fifth substrate processing unit PU5 perform substrate processing in parallel. Note that while the example shown in Figures 4A and 4B shows a time schedule in which substrate processing is performed on two priority substrates W2 in parallel, the interrupt schedule may also show a time schedule in which substrate processing is performed on a normal substrate W1 and a priority substrate W2 in parallel.

[0110] Next, the process of creating an interrupt schedule (time schedule) will be described with reference to Figures 1A to 5. Figure 5 is a diagram showing an example of the flow of the process of creating an interrupt schedule (time schedule). In more detail, Figure 5 shows the process of arranging multiple plan elements BL in a time table TB. Hereinafter, the process of arranging multiple plan elements BL in a time table TB may be referred to as the "time schedule creation process."

[0111] As shown in FIG. 5 , the processing unit 105 references the current state of the timetable TB and a plurality of plan elements BL to acquire plan elements BL that have not yet been allocated to the timetable TB (step S1). Hereinafter, unallocated plan elements BL may be referred to as "plan elements NBL." At the start of the time schedule creation process, all plan elements BL assigned to each unprocessed priority substrate W2b and all plan elements BL assigned to each container-remaining substrate W1b are the plan elements NBL. For example, under the condition that four plan elements BL are allocated to each substrate W in the timetable TB, if the total number of substrates W to be processed is 25, the processing unit 105 acquires 100 plan elements NBL at the start of the time schedule creation process. As the time schedule creation process progresses, the number of plan elements NBL decreases. The time schedule creation process continues until all plan elements NBL remain.

[0112] When the calculation processing unit 105 acquires the plan element NBL, it references the current state of the timetable TB, the plan element NBL, the constraints (constraints 1 to 9), the processing procedure PD, the processing time PT, information about the substrate processing to be performed on the normal substrate W1, and information about the substrate processing to be performed on the priority substrate W2, and acquires a plan element BL that can be placed next from among the plan elements NBL. Hereinafter, the plan element BL that can be placed next is sometimes referred to as a "plan element ABL."

[0113] When the calculation processing unit 105 acquires the plan element ABL, it refers to the current state of the timetable TB, the plan element ABL, the constraints (constraints 1 to 9), the processing procedure PD, the processing time PT, information about the substrate processing to be performed on the normal substrate W1, and information about the substrate processing to be performed on the priority substrate W2, and calculates the allocation possible times for each of the plan elements ABL (step S2). The allocation possible times correspond to the times indicated by the timetable TB. More specifically, the allocation possible times indicate the times on the timetable TB at which the processing included in the plan element ABL can be started.

[0114] After calculating the possible placement time, the calculation processing unit 105 generates an action AC for each plan element ABL (step S3). The action AC indicates an action to place the plan element ABL at the possible placement time in the timetable TB.

[0115] After generating the actions AC, the calculation processing unit 105 selects one of the action ACs (step S4), and arranges the plan element BL (plan element ABL) in the timetable TB based on the selected action AC (step S5). As a result, the timetable TB is updated. In other words, the timetable TB becomes the following state.

[0116] Here, a first example of the process of selecting an action AC will be described. For example, in the early stage of reinforcement learning, the calculation processing unit 105 randomly selects one of the action ACs. When the number of times that reinforcement learning steps (trials) are repeated reaches a second threshold, the calculation processing unit 105 predicts and selects, from among the action ACs, the action AC that will maximize the cumulative reward. Note that one step (one trial) indicates the process from the start of the time schedule creation process until all plan elements BL are placed in the time table TB and the cumulative reward is obtained.

[0117] Specifically, the first generation program GP1 and the second generation program GP2 each include a behavior selection neural network 121 (see FIG. 6). The behavior selection neural network 121 includes a neural network that constructs a predictor. The predictor calculates an evaluation value (e.g., Q value) for each action AC. The evaluation value indicates the expected value of the cumulative reward. The calculation processing unit 105 (behavior selection neural network 121) selects the action AC with the largest evaluation value (expected value of the cumulative reward).

[0118] Note that the arithmetic processing unit 105 may randomly select one of the action ACs based on a predetermined selection condition after the number of times the reinforcement learning step (trial) has been repeated reaches or exceeds a second threshold. For example, the selection condition may indicate a period for randomly selecting one of the action ACs. In this case, the arithmetic processing unit 105 randomly selects one of the action ACs periodically.

[0119] Alternatively, the selection condition may indicate the timing for randomly selecting one of the actions AC by the number of steps (number of trials). Hereinafter, the number of steps that specifies the timing for randomly selecting one of the actions AC may be referred to as the "number of random selection steps."

[0120] The random selection step count may indicate a number of steps. In this case, the calculation processing unit 105 randomly selects one of the actions AC every time the number of times the reinforcement learning steps have been repeated (the number of trials) reaches the number of steps included in the random selection step count.

[0121] Next, a second example of the process of selecting an action AC will be described. For example, the calculation processing unit 105 acquires a random number and determines whether the value of the acquired random number is equal to or greater than a third threshold. If the value of the random number is equal to or greater than the third threshold, the calculation processing unit 105 predicts and selects, from among the action ACs, an action AC that will maximize the cumulative reward. If the value of the acquired random number is less than the third threshold, the calculation processing unit 105 randomly selects one of the action ACs. The calculation processing unit 105 decreases the third threshold as the number of steps (number of trials) increases.

[0122] When the timetable TB is updated, the calculation processing unit 105 executes the processes of steps S1 to S5 again. The calculation processing unit 105 repeats the processes of steps S1 to S5 until there are no more plan elements NBL. As a result, the time schedule creation process ends.

[0123] When the time schedule creation process is completed, the calculation processing unit 105 obtains the accumulated reward based on the final state of the timetable TB and the conditions for granting the reward (step S6). As already explained, the conditions for granting the reward include the first reward condition and the second reward condition. In this embodiment, the conditions for granting the reward further include the third reward condition and the fourth reward condition.

[0124] Next, the reinforcement learning system 120 constructed by the execution of the generation program GP by the arithmetic processing unit 105 will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the reinforcement learning system 120.

[0125] As shown in FIG. 6 , the reinforcement learning system 120 selects an action for the agent AG. The action of the agent AG causes an interaction between the agent AG and the environment EB. Specifically, the action of the agent AG includes an action AC described with reference to FIG. 5 . The environment EB includes a timetable TB described with reference to FIGS. 4A and 4B . The reinforcement learning system 120 selects the action AC as described with reference to FIG. 5 . The agent AG places a plan element BL in the timetable TB based on the selected action AC.

[0126] The reinforcement learning system 120 includes an action selection neural network 121 and a training engine 122. The training engine 122 includes a replay memory 123. The replay memory 123 is included in the storage area of ​​the storage unit 102 described with reference to Figures 1A and 1B.

[0127] As already explained, the behavior selection neural network 121 calculates an evaluation value (e.g., Q value) for each action AC and selects the action AC with the largest evaluation value. The training engine 122 trains the behavior selection neural network 121 and adjusts multiple parameters (weighting coefficients) included in the behavior selection neural network 121.

[0128] More specifically, the training engine 122 stores the experience data in the replay memory 123. The experience data indicates the results of the interaction of the agent AG with the environment EB. The experience data is learning data (training data) for supervised learning, and the training engine 122 trains the action selection neural network 121 by having the action selection neural network 121 learn the experience data. As a result, multiple parameters (weighting coefficients) included in the action selection neural network 121 are adjusted, and the prediction accuracy of the action selection neural network 121 is improved.

[0129] Specifically, the training engine 122 (the calculation processing unit 105) generates experience data every time a plan element BL is placed in the timetable TB and stores the experience data in the replay memory 123. The experience data includes the current state of the timetable TB, the current plan element NBL (unassigned plan element BL), the current available time for placement, the selected action AC (selected plan element ABL), and the state of the next timetable TB. The experience data upon completion of creation of the time schedule (interrupt schedule) further includes the value of the accumulated reward.

[0130] The training engine 122 (the processing unit 105) trains the behavior selection neural network 121 based on predetermined training timing conditions. For example, the training timing conditions may indicate a period for training the behavior selection neural network 121. In this case, the training engine 122 (the processing unit 105) periodically trains the behavior selection neural network 121 by having the behavior selection neural network 121 learn experience data.

[0131] Alternatively, the training timing condition may indicate the timing for training the behavior selection neural network 121 in terms of the number of steps. Hereinafter, the number of steps that specifies the timing for training the behavior selection neural network 121 may be referred to as the "number of training steps." The number of training steps indicates a number of steps. In this case, the training engine 122 (arithmetic processing unit 105) trains the behavior selection neural network 121 by having the behavior selection neural network 121 learn empirical data each time the number of times the reinforcement learning steps have been repeated (the number of trials) reaches the number of steps included in the number of training steps.

[0132] Next, a schedule creation program generation method of this embodiment will be described with reference to Figures 1A to 10. Figure 7 is a flowchart showing the schedule creation program generation method of this embodiment. The schedule creation program generation method of this embodiment is executed by the schedule creation program generation device 100 described with reference to Figures 1A to 6. Specifically, the schedule creation program generation method of this embodiment is executed by the calculation processing unit 105 described with reference to Figures 1A to 6. Therefore, Figure 7 shows the flow of processing executed by the calculation processing unit 105.

[0133] 1A to 10, a method for creating a first schedule creation program CP1 based on a first generation program GP1 will be described. The first schedule creation program CP1 is a computer program for creating a time schedule (interrupt schedule) for processing the normal substrates W1b (container-remaining substrates W1b) remaining in the first substrate accommodation container CA1, which has been placed earlier on the container platform LPS, and the plurality of substrates W (priority substrates W2) that constitute the priority lot, in response to a substrate accommodation container CA (second substrate accommodation container CA2) that accommodates a priority lot being placed on the container platform LPS, when some of the substrates W (normal substrates W1b) that constitute a normal lot remain in the first substrate accommodation container CA1, which has been placed earlier on the container platform LPS.

[0134] 7, the schedule creation program generation method of the present embodiment includes steps S11 to S13. For example, the process shown in FIG. 7 may be started in response to an instruction to start reinforcement learning by the first generation program GP1 by operating the input unit 101 by an operator with the first generation program GP1 installed in the schedule creation program generation device 100.

[0135] 7 starts, the calculation processing unit 105 executes a time schedule creation process to create an interrupt schedule (time schedule) (step S11). The time schedule creation process (processing for creating an interrupt schedule) will be described later with reference to FIG.

[0136] After creating the interrupt schedule, the calculation processing unit 105 determines the reward to be awarded for the final state of the timetable TB based on the final state of the timetable TB and the conditions for awarding the reward, and obtains the accumulated reward (step S12). The final state of the timetable TB represents the created interrupt schedule. As already explained, the final state of the timetable TB represents the timetable TB in which all of the plan elements BL have been arranged. All of the plan elements BL include multiple plan elements BL (first plan element BL1 to fourth plan element BL4) assigned to each of the container-remaining substrates W1b and multiple plan elements BL (first plan element BL1 to fourth plan element BL4) assigned to each of the priority substrates W2b. Step S12 is an example of a "reward determination step."

[0137] When creating the first schedule creation program CP1 based on the first generation program GP1, the calculation processing unit 105 acquires the first to fourth reward conditions as conditions for granting rewards. The first to fourth reward conditions may be input by an operator operating the input unit 101 before the start of reinforcement learning using the first generation program GP1, or may be included in the first generation program GP1.

[0138] As already explained, the first remuneration condition indicates that the first remuneration is to be awarded when the final state of the timetable TB indicates that processing of at least one of the container remaining substrates W1b is completed before processing of the priority lot is completed. The second remuneration condition indicates that the second remuneration is to be awarded when the final state of the timetable TB indicates that processing of the priority lot is completed before processing of the regular lot is completed. The third remuneration condition indicates that the third remuneration, which has a larger value, is to be awarded the earlier the time at which processing of the priority lot is completed. The fourth remuneration condition indicates that the fourth remuneration, which has a larger value, is to be awarded the earlier the time at which processing of the regular lot is completed.

[0139] After creating the interruption schedule and obtaining the cumulative reward, the calculation processing unit 105 determines whether to end the reinforcement learning (step S13). If the calculation processing unit 105 determines not to end the reinforcement learning (No in step S13), the process shown in Fig. 7 returns to the process of step S11. On the other hand, if the calculation processing unit 105 determines to end the reinforcement learning (Yes in step S13), the process shown in Fig. 7 ends. As a result, a first schedule creation program CP1 (schedule creation program CP) is created.

[0140] 1A and 1B, a first threshold may be set as a condition for terminating reinforcement learning. In this case, the arithmetic processing unit 105 determines to terminate reinforcement learning when the number of times the reinforcement learning steps have been repeated (the number of trials) becomes equal to or greater than the first threshold.

[0141] As described with reference to Figures 5 and 6, the calculation processing unit 105 adjusts the parameters (weighting coefficients) of the action selection neural network 121 in the process of repeating the time schedule creation process. Therefore, as the reinforcement learning steps (trials) are repeated, the value of the final reward increases. In this embodiment, one reinforcement learning step (one trial) includes steps S11 to S13 shown in Figure 7. Note that, among steps S11 to S13, the step including steps S11 and S12 is an example of an "experience step."

[0142] Next, the time schedule creation process (step S11) shown in Fig. 7 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the time schedule creation process. As shown in Fig. 8, the time schedule creation process includes steps S21 to S25.

[0143] When the calculation processing unit 105 starts the time schedule creation process, it determines the normal lot conditions (step S21). When the first schedule creation program CP1 is created based on the first generation program GP1, the normal lot conditions include the number of container-remaining substrates W1b, the number of normal substrates W1 remaining (positioned) inside the substrate processing apparatus WP, and information about the substrate processing to be performed on the normal lot (the substrate processing procedure and the processing conditions for that substrate processing). For example, the calculation processing unit 105 may randomly generate information about the substrate processing to be performed on the normal lot. Hereinafter, the normal substrates W1 remaining (positioned) inside the substrate processing apparatus WP may be referred to as "apparatus-remaining substrates W1."

[0144] The respective numbers of container residual substrates W1b and apparatus residual substrates W1 may be generated randomly by the arithmetic processing unit 105 within a range in which the total number of container residual substrates W1b and apparatus residual substrates W1 is, for example, 25 or less. Alternatively, the respective numbers of container residual substrates W1b and apparatus residual substrates W1 may be selected randomly by the arithmetic processing unit 105 from among combination patterns of the numbers of container residual substrates W1b and apparatus residual substrates W1. Information indicating combination patterns of the numbers of container residual substrates W1b and apparatus residual substrates W1 may be included in advance in the first generation program GP1, or may be input by the operator operating the input unit 101.

[0145] After determining the normal lot conditions, the arithmetic processing unit 105 determines priority lot conditions (step S22). The priority lot conditions indicate the number of unprocessed priority substrates W2b and information regarding the substrate processing to be performed on the priority lot (the procedure for the substrate processing and the processing conditions for the substrate processing). The arithmetic processing unit 105 may randomly generate the number of priority substrates W2b and the information regarding the substrate processing to be performed on the priority lot.

[0146] After determining the priority lot conditions, the calculation processing unit 105 obtains a timetable TB (step S23). Specifically, when creating the first schedule creation program CP1 based on the first generation program GP1, the calculation processing unit 105 creates a timetable TB in which plan elements BL assigned to each normal substrate W1 (apparatus remaining substrates W1) remaining (positioned) inside the substrate processing apparatus WP are arranged. Note that information on a timetable TB in which no plan elements BL are arranged (an empty timetable TB) may be included in the generation program GP or may be stored in advance in the storage unit 102.

[0147] More specifically, the processing unit 105 acquires multiple plan elements BL (first plan element BL1 to fourth plan element BL4) for each remaining substrate W1 in the equipment by referring to the information indicating the number of remaining substrates W1 in the equipment determined in step S21. Then, the processing unit 105 randomly arranges the multiple plan elements BL assigned to each remaining substrate W1 in the time table TB by referring to the constraints (constraints 1 to 9) shown in Table 1 and information related to substrate processing to be performed on a normal lot.

[0148] More specifically, when some of the substrates W (normal substrates W1) that constitute a normal lot remain in the first substrate storage container CA1 that is placed prior to the container loading section LPS, and a substrate storage container CA (second substrate storage container CA2) that stores a priority lot is placed on the container loading section LPS, the other normal substrates W1 among the substrates W that constitute the normal lot that do not remain in the first substrate storage container CA1 are located (remain) inside the substrate processing apparatus WP or are transported into the third substrate storage container CA3.

[0149] A normal substrate W1 (device-remaining substrate W1) located (remaining) inside the substrate processing apparatus WP corresponds to either a processed normal substrate W1a before being loaded into the third substrate storage container CA3, an unprocessed normal substrate W1b loaded out of the first substrate storage container CA1, or a normal substrate W loaded into the substrate processing unit PU.

[0150] The processed normal substrate W1a before being loaded into the third substrate accommodation container CA3 refers to a normal substrate W1 that has undergone substrate processing and been unloaded from the substrate processing unit PU but has not been loaded into the third substrate accommodation container CA3. The unprocessed normal substrate W1b unloaded from the first substrate accommodation container CA1 refers to a normal substrate W1 that has been unloaded from the first substrate accommodation container CA1 but has not been loaded into the substrate processing unit PU. The normal substrate W loaded into the substrate processing unit PU refers to a normal substrate W1 that is currently undergoing substrate processing by the substrate processing unit PU.

[0151] The calculation processing unit 105 randomly selects the planning elements BL assigned to each of the remaining substrates W1 in the equipment and places them on the timetable TB, by referring to the constraints (constraints 1 to 9) shown in Table 1 and information regarding the substrate processing to be performed on normal lots, so that the state of the remaining substrates W1 in the equipment becomes one of the three states described above.

[0152] Steps S21 to S23 are executed each time the reinforcement learning step (trial) is repeated. That is, each time the reinforcement learning step (trial) is repeated, the process of determining normal lot conditions, the process of determining priority lot conditions, and the process of obtaining the timetable TB are repeated.

[0153] After acquiring the timetable TB, the calculation processing unit 105 sequentially arranges the plurality of plan elements BL in the timetable TB based on the information on the number of remaining substrates W1b in the container, information on the substrate processing to be performed on the normal lot, information on the number of priority substrates W2b, information on the substrate processing to be performed on the priority lot, information indicating the processing procedure PD, information indicating the processing time PT, information indicating the plan elements BL, and the constraints (constraints 1 to 9) shown in Table 1, thereby sequentially changing the state of the timetable TB (steps S24 and S25). Steps S24 and S25 are examples of an "arrangement step."

[0154] Specifically, the processing unit 105 refers to the information on the number of remaining substrates W1 b in the container to obtain a plurality of plan elements BL to be assigned to each of the remaining substrates W1 b in the container, and refers to the information on the number of priority substrates W2 to obtain a plurality of plan elements BL to be assigned to each of the priority substrates W2 b. The processing unit 105 refers to the information on the substrate processing to be performed on the normal lot, the information on the substrate processing to be performed on the priority lot, and the constraints (constraints 1 to 9) shown in Table 1, and places one of the plurality of plan elements BL on the time table TB to change the state of the time table TB (step S24).

[0155] After arranging one of the plurality of plan elements BL in the timetable TB, the calculation processing unit 105 determines whether or not all of the plan elements BL have been arranged in the timetable TB (step S25). In other words, the calculation processing unit 105 determines whether or not an interrupt schedule (time schedule) has been created.

[0156] When the calculation processing unit 105 determines that all the plan elements BL have been arranged in the timetable TB (Yes in step S25), the time schedule creation process ends, and the calculation processing unit 105 determines the reward (step S12 in FIG. 7). Steps S21 to S25 are executed each time a reinforcement learning step (trial) is repeated. In other words, an interrupt schedule (time schedule) is created each time a reinforcement learning step (trial) is repeated.

[0157] If the calculation processing unit 105 determines that some of the multiple plan elements BL are not arranged in the timetable TB (No in step S25), the calculation processing unit 105 returns to step S24, selects the next plan element BL, and arranges it in the timetable TB. In this way, the calculation processing unit 105 arranges the multiple plan elements BL in the timetable TB sequentially, and sequentially changes the state of the timetable TB.

[0158] Next, the process of placing the plan elements BL in the timetable TB (step S24 in FIG. 8) will be described with reference to FIGS. 9 and 10. As described with reference to FIGS. 5 and 6, the calculation processing unit 105 randomly selects one of the actions AC, thereby placing one of the plan elements BL (plan elements ABL) that can be placed next in the timetable TB. Alternatively, the calculation processing unit 105 predicts and selects, from among the action ACs, the action AC that will maximize the reward, thereby placing one of the plan elements BL (plan elements ABL) that can be placed next in the timetable TB.

[0159] 9 is a flowchart showing a process of randomly selecting one of the actions AC and arranging the plan element BL in the timetable TB. As shown in FIG. 9, when randomly selecting one of the actions AC, the process of arranging the plan element BL in the timetable TB (step S24 in FIG. 8) includes steps S31 to S34.

[0160] When the calculation processing unit 105 starts the process of placing the plan element BL on the timetable TB, it acquires the plan element BL (plan element ABL) that can be placed next on the timetable TB from among the unplaced plan elements BL (plan element NBL) (step S31).

[0161] When the calculation processing unit 105 acquires the next plan element BL (plan element ABL) that can be placed, it acquires the placement possible time of each plan element ABL (step S32). After acquiring the placement possible time, the calculation processing unit 105 randomly selects one of the plan elements ABL (step S33). More specifically, the calculation processing unit 105 generates an action AC for each plan element ABL and randomly selects one of the action ACs.

[0162] The calculation processing unit 105 arranges the randomly selected plan element ABL in the timetable TB (step S34). As a result, the processing shown in Fig. 9 ends. In detail, the calculation processing unit 105 arranges the plan element BL (plan element ABL) in the timetable TB based on the randomly selected action AC.

[0163] 10 is a flowchart showing a process of predicting an action AC that will maximize the reward and arranging the plan element BL in the timetable TB. As shown in FIG. 10, when predicting an action AC that will maximize the reward, the process of arranging the plan element BL in the timetable TB (step S24 in FIG. 8) includes steps S41 to S44.

[0164] Steps S41, S42, and S44 shown in FIG. 10 are the same as steps S31, S32, and S34 shown in FIG. 9, and therefore their description will be omitted.

[0165] As shown in Fig. 10, after acquiring the available placement time, the processing unit 105 selects the plan element BL (plan element ABL) with the largest evaluation value (expected reward value) (step S43). More specifically, the processing unit 105 generates an action AC for each plan element ABL. Then, using the behavior selection neural network 121 (see Fig. 6), the processing unit 105 selects the action AC with the largest evaluation value from among the action ACs. The processing unit 105 places the plan element BL (plan element ABL) in the timetable TB based on the selected action AC.

[0166] 1 to 10, a method for creating the second schedule creation program CP2 based on the second generation program GP2 will be described. The second schedule creation program CP2 is a computer program that creates a time schedule (interrupt schedule) for processing, by the substrate processing apparatus WP, the normal substrates W1 (container-remaining substrates W1b) remaining in the first substrate accommodation container CA1 and the plurality of substrates W (priority substrates W2) constituting the priority lot, in response to a substrate accommodation container CA (second substrate accommodation container CA2) being placed on the container platform LPS when all of the substrates W (normal substrates W1) constituting a normal lot remain in the first substrate accommodation container CA1, which is placed previously on the container platform LPS.

[0167] The method of creating the second schedule creation program CP2 based on the second generation program GP2 differs from the method of creating the first schedule creation program CP1 based on the first generation program GP1 in terms of the first remuneration conditions and the normal lot conditions.

[0168] When the second schedule creation program CP2 is created based on the second creation program GP2, the first remuneration condition indicates that the first remuneration is to be awarded when the final state of the timetable TB indicates that processing of a predetermined number of container-remaining substrates W1b has been completed before processing of a priority lot is completed. The predetermined number is set based on the number of priority substrates W2. For example, the predetermined number may be set to half the number of substrates W that make up the priority lot. Information for setting the predetermined number may be included in the second creation program GP2 in advance, or may be input by the operator operating the input unit 101.

[0169] Furthermore, when the second schedule creation program CP2 is created based on the second creation program GP2, the normal lot conditions include the number of container remaining substrates W1b and information about the substrate processing to be performed on the normal lot, but do not include the number of apparatus remaining substrates W1. Therefore, when the second schedule creation program CP2 is created based on the second creation program GP2, the calculation processing unit 105 obtains a timetable TB in which no plan element BL is placed (an empty timetable TB) in step S23 of FIG.

[0170] Next, the substrate processing apparatus 200 of this embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram showing a substrate processing system 200A including the substrate processing apparatus 200 of this embodiment.

[0171] 11, the substrate processing system 200A includes a recording medium 110 and a substrate processing apparatus 200. The substrate processing apparatus 200 includes an interface unit 201, a container placement unit LPS, a transport unit TR, a plurality of substrate processing units PU, a memory unit 202, and a control unit 203a.

[0172] In this embodiment, the container platform LPS has multiple load ports LP. For example, as described with reference to FIG. 2, the container platform LPS may have four load ports LP (first load port LP1 to fourth load port LP4). Furthermore, as described with reference to FIG. 2, the transport unit TR may have an indexer robot IR, a transfer unit PS (not shown), and a transport robot CR. The substrate processing apparatus 200 may also have 16 substrate processing units PU. The configurations of the container platform LPS, transport unit TR, and substrate processing unit PU are similar to those of the container platform LPS, transport unit TR, and substrate processing unit PU described with reference to FIG. 2, and therefore, description thereof will be omitted.

[0173] The interface unit 201 exchanges information, data, or signals with the recording medium 110. Specifically, the recording medium 110 stores the schedule creation programs CP (first schedule creation program CP1 and second schedule creation program CP2) described with reference to FIGS. 1A to 10. The interface unit 201 reads the schedule creation programs CP from the recording medium 110 and inputs them to the control unit 203a. The configuration of the interface unit 201 is similar to that of the interface unit 103 described with reference to FIGS. 1A and 1B, and therefore a detailed description thereof will be omitted.

[0174] The memory unit 202 stores various types of information for controlling the operation of the substrate processing apparatus 200. For example, the memory unit 202 stores data and computer programs. The data includes various types of recipe data. The recipe data includes, for example, a process recipe. The process recipe is data that defines a procedure for substrate processing. The memory unit 202 also stores schedule creation programs CP (first schedule creation program CP1 and second schedule creation program CP2) read from the recording medium 110. The memory unit 202 further stores a processing procedure PD, a processing time PT, a planning element BL, and constraints.

[0175] The storage unit 202 includes a main storage device. The main storage device is, for example, a semiconductor memory. The storage unit 202 may further include an auxiliary storage device. The auxiliary storage device includes, for example, at least one of a semiconductor memory and a hard disk drive. The storage unit 202 may also include removable media.

[0176] The control unit 203a includes, for example, a processor. The control unit 203a may include a CPU or an MPU as the processor. Alternatively, the control unit 203a may include a general-purpose computing device or a dedicated computing device. The control unit 203a controls the operation of each unit of the substrate processing apparatus 200 based on various information stored in the memory unit 202. For example, the control unit 203a controls the interface unit 201, the load port LP, the indexer robot IR, the transport robot CR, the substrate processing unit PU, and the memory unit 202.

[0177] When the substrate accommodation container CA is placed on the container mounting part LPS, the control unit 203a acquires priority information, number information, recipe information, and processing condition information of the substrates W (lot) accommodated in the substrate accommodation container CA from a host computer (not shown), and stores the information in the memory unit 202. Here, the priority information indicates whether the substrates W accommodated in the substrate accommodation container CA are normal substrates W1 or priority substrates W2. The number information indicates the number of substrates W accommodated in the substrate accommodation container CA. The recipe information indicates information for identifying (specifying) the recipe. The control unit 203a acquires the procedure for substrate processing by referring to the recipe information. The processing condition information indicates the processing conditions for substrate processing.

[0178] Furthermore, when a substrate storage container CA (first substrate storage container CA1) that stores a normal substrate W1b is placed on the container platform LPS, the control unit 203a executes the first schedule creation program CP1 or the second schedule creation program CP2 in response to a substrate storage container CA (second substrate storage container CA2) that stores a priority substrate W2b being placed on the container platform LPS, to create an interrupt schedule. Then, the control unit 203a controls the load port LP, the indexer robot IR, the transport robot CR, and the substrate processing unit PU based on the created interrupt schedule.

[0179] Next, the processing executed by the control unit 203a will be described with reference to Fig. 11 to Fig. 13. Fig. 12 is a diagram showing the flow of processing executed by the control unit 203a included in the substrate processing apparatus 200 of this embodiment. In detail, Fig. 12 shows the flow of processing executed by the control unit 203a when creating an interrupt schedule.

[0180] The process shown in Figure 12 begins when a substrate storage container CA (first substrate storage container CA1) that stores a normal substrate W1b is placed on the container loading part LPS and a substrate storage container CA (second substrate storage container CA2) that stores a priority substrate W2b is placed on the container loading part LPS.

[0181] 12 starts, the control unit 203a determines whether all of the substrates W (normal substrates W1) that constitute the normal lot are accommodated in the first substrate accommodation container CA1 (step S101). For example, the control unit 203a may determine whether all of the substrates W (normal substrates W1) that constitute the normal lot are accommodated in the first substrate accommodation container CA1 by referring to a time schedule of the process being performed on the normal lot and the current time.

[0182] When the control unit 203a determines that all of the substrates W (normal substrates W1) constituting the normal lot are not accommodated in the first substrate accommodation container CA1 (Yes in step S101), the control unit 203a executes the first schedule creation program CP1 to create an interrupt schedule (step S103). As a result, the process shown in Fig. 12 ends. Hereinafter, the interrupt schedule created by executing the first schedule creation program CP1 may be referred to as the "first interrupt schedule."

[0183] Furthermore, when the control unit 203a determines that all of the substrates W (normal substrates W1) constituting the normal lot are accommodated in the first substrate accommodation container CA1 (Yes in step S101), it executes the second schedule creation program CP2 to create an interrupt schedule (step S105). As a result, the process shown in Fig. 12 ends. Hereinafter, the interrupt schedule created by executing the second schedule creation program CP2 may be referred to as a "second interrupt schedule."

[0184] Next, the process of creating the first interrupt schedule (step S103) will be described with reference to Fig. 13. Fig. 13 is a diagram showing the flow of the process of creating the interrupt schedule.

[0185] 13 starts, the control unit 203a acquires a timetable TB (step S111). Specifically, when creating a first interrupt schedule, the control unit 203a creates a timetable TB in which a plan element BL for the remaining substrate W1 is placed. For example, the control unit 203a may create the timetable TB by referring to the time schedule for the normal lot currently being processed. Note that information about a timetable TB in which no plan element BL is placed (an empty timetable TB) may be included in the first schedule creation program CP1 or may be stored in advance in the storage unit 202.

[0186] When the control unit 203a acquires (creates) the time table TB, it executes the first schedule creation program CP1 to create a first interruption time schedule (step S112), thereby completing the process shown in FIG.

[0187] Specifically, the first schedule creation program CP1 includes the behavior selection neural network 121 in which parameters (weighting coefficients) have been adjusted, as described with reference to Figures 5 and 6. The control unit 203a creates a time schedule (first interruption schedule) in accordance with steps S1 to S5 described with reference to Figure 5, based on the first schedule creation program CP1, regular lot information, priority lot information, processing procedure PD, processing time PT, planning elements BL, and constraint conditions.

[0188] Here, the normal lot information indicates the number of remaining substrates W1b in the container and information about the substrate processing being performed on the normal lot (the substrate processing procedure and the processing conditions for that substrate processing), while the priority lot information indicates the number of priority substrates W2b and information about the substrate processing being performed on the priority lot (the substrate processing procedure and the processing conditions for that substrate processing).

[0189] Next, the process of creating a second interrupt schedule (step S105) will be described with reference to Figure 13. The process of creating a second interrupt schedule differs from the process of creating a first interrupt schedule in step S111. Specifically, when creating a second interrupt schedule, the control unit 203a obtains a timetable TB in which no plan elements BL are placed (an empty timetable TB). Note that information about the empty timetable TB may be included in the second schedule creation program CP2 or may be stored in advance in the storage unit 202.

[0190] Note that, after the processing for the priority lot is completed, if an interruption to the processing for another priority lot occurs, the control unit 203a again executes the processing described with reference to Figures 12 and 13. Therefore, even if a substrate accommodation container CA accommodating a normal lot is placed on the container mounting part LPS, and then a plurality of substrate accommodation containers CA accommodating priority lots are successively placed on the container mounting part LPS, the processing for the priority lot (subsequent lot) can be completed before the processing for the normal lot (preceding lot) without stopping the processing for the normal lot (preceding lot).

[0191] 1A to 13, the first embodiment of the present invention has been described. According to the first embodiment, even if a priority lot interrupts a substrate accommodation container CA that accommodates a normal lot while the substrate accommodation container CA is placed on the container platform LPS first, the processing of the priority lot (following lot) can be completed before the normal lot (leading lot) without stopping the processing of the normal lot (leading lot). Therefore, the following lot, which should be processed with priority, and the normal lot that is being processed first can be processed more efficiently.

[0192] 12 and 13, the second interrupt schedule is created only when all of the regular substrates W1b that make up a regular lot are accommodated in the first substrate accommodation container CA1, but the process of the control unit 203a is not limited to this. For example, after creating the second interrupt schedule and completing the process for the priority lot, the control unit 203a may create the second interrupt schedule again if an interruption to process another priority lot occurs.

[0193] Furthermore, in this embodiment, the first remuneration condition referenced when creating the first schedule creation program CP1 indicates that the first remuneration is granted when the final state of the time table TB indicates that processing of at least one of the container-remaining substrates W1b is completed before processing of the priority lot is completed. However, the first remuneration condition referenced when creating the first schedule creation program CP1 may indicate that the first remuneration is granted when the final state of the time table TB indicates that substrate processing of a normal substrate W1 is being performed in parallel with substrate processing of at least one priority substrate W2. This condition makes it possible to create a time schedule (interrupt schedule) in which substrate processing of the priority substrate W2 and substrate processing of the normal substrate W1 are performed in parallel. This makes it possible to more reliably create a time schedule (interrupt schedule) in which processing of the priority lot and processing of the normal lot are completed in a shorter period of time.

[0194] Alternatively, the conditions for awarding the reward may include a fifth reward condition that awards the fifth reward when the final state of the timetable TB indicates a state in which substrate processing for a normal substrate W1 is being carried out in parallel with substrate processing for at least one priority substrate W2.

[0195] Similarly, the first compensation condition referenced when creating the second schedule creation program CP2 indicates that the first compensation is granted when the final state of the timetable TB indicates a state in which processing for a predetermined number of remaining substrates W1b in the container is completed before processing for the priority lot is completed, but the first compensation condition referenced when creating the second schedule creation program CP2 may also indicate that the first compensation is granted when the final state of the timetable TB indicates a state in which substrate processing for remaining substrates W1b in the container is being performed in parallel with substrate processing for a predetermined number of priority substrates W2.

[0196] Alternatively, the conditions for awarding a remuneration may include a fifth remuneration condition under which a fifth remuneration is awarded when the final state of the timetable TB indicates a state in which substrate processing for a normal substrate W1 is being performed in parallel with substrate processing for at least one priority substrate W2.Furthermore, the conditions for awarding a remuneration may include a sixth remuneration condition under which a sixth remuneration is awarded when the final state of the timetable TB indicates a state in which substrate processing for a normal substrate W1 is being performed in parallel with substrate processing for a predetermined number of priority substrates W2.

[0197] [Embodiment 2] Next, a second embodiment of the present invention will be described with reference to Fig. 14. However, differences from the first embodiment will be described, and a description of the same aspects as in the first embodiment will be omitted. In the second embodiment, unlike the first embodiment, the substrate processing apparatus 200 generates a schedule creation program CP (a first schedule creation program CP1 and a second schedule creation program CP2).

[0198] Fig. 14 is a diagram showing a substrate processing system 200A including the substrate processing apparatus 200 of this embodiment. As shown in Fig. 14, the substrate processing system 200A includes a recording medium 110 and the substrate processing apparatus 200. The substrate processing apparatus 200 includes a container mounting part LPS, a transport part TR, a plurality of substrate processing units PU, a memory part 202, and a control part 203b.

[0199] The recording medium 110 stores the generation programs GP (first generation program GP1 and second generation program GP2) described with reference to Figures 1A to 13. The interface unit 201 of the substrate processing apparatus 200 reads the generation programs GP from the recording medium 110 and inputs them to the control unit 203b.

[0200] The storage unit 202 stores the generation program GP read from the recording medium 110. The storage unit 202 also stores recipe data, processing procedures PD, processing times PT, plan elements BL, and constraint conditions, as described with reference to FIG.

[0201] The control unit 203b has a processor. The processor of the control unit 203b may be a CPU, an MPU, a GPU, an NPU, or a quantum computer. Alternatively, the control unit 203b may have a general-purpose computing device or a dedicated computing device. The control unit 203b controls the operation of each unit of the substrate processing apparatus 200 based on various information stored in the memory unit 202. For example, the control unit 203b controls the interface unit 201, the load port LP, the indexer robot IR, the transport robot CR, the substrate processing unit PU, and the memory unit 202.

[0202] 1A to 10, the control unit 203b executes the first generation program GP1 to generate the first schedule creation program CP1, and executes the second generation program GP2 to generate the second schedule creation program CP2. The schedule creation programs CP (the first schedule creation program CP1 and the second schedule creation program CP2) are stored in the storage unit 202.

[0203] In addition, similar to the control unit 203a described with reference to Figures 11 to 13, when the substrate storage container CA is placed on the container loading part LPS, the control unit 203b acquires priority information, number information, recipe information, and processing condition information of the substrates W stored in the substrate storage container CA from a host computer not shown, and stores the information in the memory unit 202.

[0204] 11 to 13, the control unit 203b executes the first schedule creation program CP1 or the second schedule creation program CP2 to create an interrupt schedule in response to a substrate storage container CA (second substrate storage container CA2) storing a priority substrate W2b being placed on the container platform LPS while a substrate storage container CA (first substrate storage container CA1) storing a normal substrate W1b is placed on the container platform LPS. Then, the control unit 203b controls the load port LP, the indexer robot IR, the transport robot CR, and the substrate processing unit PU based on the created interrupt schedule.

[0205] The second embodiment of the present invention has been described above with reference to Fig. 14. According to the second embodiment, similar to the first embodiment, processing of a priority lot (a subsequent lot) can be completed before processing of a normal lot (a preceding lot) without stopping the processing of the normal lot (a preceding lot). Therefore, the subsequent lot that should be processed with priority and the normal lot that is being processed in advance can be processed more efficiently.

[0206] [Embodiment 3] Next, a third embodiment of the present invention will be described with reference to Fig. 15. However, differences from the first and second embodiments will be described, and a description of the same aspects as the first and second embodiments will be omitted. Unlike the first and second embodiments, the third embodiment is configured such that a substrate processing system 200B includes a schedule creation device 300.

[0207] 15 is a diagram showing a substrate processing system 200B of this embodiment. As shown in FIG. 15, the substrate processing system 200B includes a schedule creation device 300 and a substrate processing apparatus 200.

[0208] The schedule creation device 300 creates an interrupt schedule based on a schedule creation program CP (a first schedule creation program CP1 and a second schedule creation program CP2). Specifically, the schedule creation device 300 includes a storage unit 302, a communication unit 303, and a processing unit 304. The schedule creation device 300 is, for example, a server.

[0209] The storage unit 302 has a main storage device. The main storage device includes, for example, a semiconductor memory. The storage unit 302 may further include an auxiliary storage device. The auxiliary storage device includes, for example, at least one of a semiconductor memory and a hard disk drive. The storage unit 302 may include removable media. The storage unit 302 stores various computer programs and various data. Specifically, the storage unit 302 stores a schedule creation program CP (a first schedule creation program CP1 and a second schedule creation program CP2). As described with reference to FIGS. 1A to 10 , the schedule creation program CP (the first schedule creation program CP1 and the second schedule creation program CP2) is generated based on the generation program GP (the first generation program GP1 and the second generation program GP2).

[0210] The arithmetic processing unit 304 includes a processor. The arithmetic processing unit 304 may include a CPU or an MPU as the processor. Alternatively, the arithmetic processing unit 304 may include a general-purpose arithmetic device or a dedicated arithmetic device. Similar to the control unit 203a described with reference to FIGS. 11 to 13, the arithmetic processing unit 304 executes the schedule creation program CP stored in the storage unit 302 to create an interrupt schedule.

[0211] The communication unit 303 is connected to a network and performs communication with the substrate processing apparatus 200. The network includes, for example, the Internet, a local area network (LAN), a public telephone network, and a short-range wireless network. The communication unit 303 includes a communication device. The communication unit 303 is, for example, a network interface controller.

[0212] The communication unit 303 is controlled by the arithmetic processing unit 304 to receive regular lot information and priority lot information from the substrate processing apparatus 200. The communication unit 303 is also controlled by the arithmetic processing unit 304 to transmit an interrupt schedule created by the arithmetic processing unit 304 to the substrate processing apparatus 200. The communication unit 303 is an example of a "transmitting unit."

[0213] The substrate processing apparatus 200 includes a container platform LPS, a transport unit TR, a plurality of substrate processing units PU, a storage unit 202, a control unit 203c, and a communication unit 204.

[0214] The communication unit 204 is connected to a network and communicates with the communication unit 303 of the schedule creation device 300. The communication unit 204 includes a communication device. The communication unit 204 is, for example, a network interface controller. The communication unit 204 is controlled by the control unit 203c to transmit regular lot information and priority lot information to the schedule creation device 300. The communication unit 204 is also controlled by the control unit 203c to receive the interrupt schedule transmitted from the communication unit 303 of the schedule creation device 300. The communication unit 204 is an example of a "receiving unit."

[0215] The control unit 203c has, for example, a processor. The control unit 203c may have a CPU or an MPU as the processor. Alternatively, the control unit 203c may have a general-purpose computing device or a dedicated computing device. The control unit 203c controls the operation of each unit of the substrate processing apparatus 200 based on various information stored in the memory unit 202. For example, the control unit 203c controls the load port LP, the indexer robot IR, the transport robot CR, the substrate processing unit PU, the memory unit 202, and the communication unit 204.

[0216] Similar to the control unit 203a described with reference to Figures 11 to 13, when the substrate storage container CA is placed on the container loading part LPS, the control unit 203c acquires priority information, number information, recipe information, and processing condition information of the substrates W stored in the substrate storage container CA from a host computer (not shown) and stores them in the memory unit 202.

[0217] Furthermore, when a substrate accommodation container CA (first substrate accommodation container CA1) that accommodates a normal substrate W1b is placed on the container platform LPS, the control unit 203c instructs the schedule creation device 300 to create an interruption schedule in response to a substrate accommodation container CA (second substrate accommodation container CA2) that accommodates a priority substrate W2b being placed on the container platform LPS. As a result, the communication unit 204 receives the interruption schedule from the schedule creation device 300. The control unit 203c controls the load port LP, the indexer robot IR, the transport robot CR, and the substrate processing unit PU based on the interruption schedule received by the communication unit 204.

[0218] Specifically, the control unit 203c causes the communication unit 204 to transmit a command to create an interrupt schedule to the schedule creation device 300. As a result, the communication unit 204 receives the interrupt schedule from the schedule creation device 300. The command to create the interrupt schedule includes regular lot information and priority lot information. Note that the regular lot information further includes information indicating whether all of the regular substrates W1b that make up the regular lot are accommodated in the first substrate accommodation container CA1.

[0219] For example, when the schedule creation device 300 (communication unit 303) and the substrate processing apparatus 200 (communication unit 204) are communicatively connected, the control unit 203c may cause the communication unit 204 to transmit information indicating the processing procedure PD, information indicating the processing time PT, information indicating the plan elements BL, and the constraints to the schedule creation device 300. Alternatively, the information indicating the processing procedure PD, information indicating the processing time PT, information indicating the plan elements BL, and the constraints may be stored in advance in the storage unit 302 of the schedule creation device 300.

[0220] The third embodiment of the present invention has been described above with reference to Fig. 15. According to the third embodiment, similar to the first and second embodiments, processing of a priority lot (a subsequent lot) can be completed before processing of a normal lot (a preceding lot) without stopping the processing of the normal lot (a preceding lot). Therefore, the subsequent lot that should be processed with priority and the normal lot that is being processed in advance can be processed more efficiently.

[0221] [Fourth Embodiment] Next, a fourth embodiment of the present invention will be described with reference to Fig. 16. However, differences from the first to third embodiments will be described, and a description of the same aspects as the first to third embodiments will be omitted. Unlike the first to third embodiments, the fourth embodiment is such that a substrate processing system 200C includes a schedule creation program generation device 100.

[0222] Fig. 16 is a diagram showing a substrate processing system 200C of this embodiment. As shown in Fig. 16, the substrate processing system 200C includes a schedule creation program generating device 100 and a substrate processing apparatus 200. The schedule creation program generating device 100 is, for example, a server.

[0223] The schedule creation program generation device 100 includes an input unit 101, a storage unit 102, a calculation processing unit 105, and a communication unit 106. As described with reference to FIGS. 1A to 10, the calculation processing unit 105 executes the generation program GP (first generation program GP1 and second generation program GP2) stored in the storage unit 102 to generate the schedule creation program CP (first schedule creation program CP1 and second schedule creation program CP2).

[0224] The communication unit 106 is connected to a network and communicates with the substrate processing apparatus 200. The communication unit 106 is controlled by the arithmetic processing unit 105 and transmits the schedule creation program CP (first schedule creation program CP1 and second schedule creation program CP2) created by the arithmetic processing unit 105 to the substrate processing apparatus 200. The communication unit 106 is an example of a "transmission unit." Note that the configuration of the communication unit 106 is similar to that of the communication unit 303 described with reference to FIG. 15 , and therefore a description thereof will be omitted.

[0225] The substrate processing apparatus 200 includes a container platform LPS, a transport unit TR, a plurality of substrate processing units PU, a storage unit 202 , a control unit 203 a , and a communication unit 204 .

[0226] The communication unit 204 is connected to a network and communicates with the communication unit 106 of the schedule creation program generation device 100. The communication unit 204 receives the schedule creation programs CP (first schedule creation program CP1 and second schedule creation program CP2) transmitted from the communication unit 106 of the schedule creation program generation device 100. The schedule creation programs CP received by the communication unit 204 are stored in the storage unit 202.

[0227] When a substrate storage container CA (second substrate storage container CA2) accommodating a priority substrate W2b is placed on the container platform LPS while a substrate storage container CA (first substrate storage container CA1) accommodating a normal substrate W1b is placed on the container platform LPS, the control unit 203a executes the schedule creation program CP to create an interrupt schedule. Then, the control unit 203a controls the load port LP, the indexer robot IR, the transport robot CR, and the substrate processing unit PU based on the created interrupt schedule.

[0228] The fourth embodiment of the present invention has been described above with reference to Fig. 16. According to the fourth embodiment, similar to the first to third embodiments, processing of a priority lot (a subsequent lot) can be completed before processing of a normal lot (a preceding lot) without stopping the processing of the normal lot (a preceding lot). Therefore, the subsequent lot that should be processed with priority and the normal lot that is being processed in advance can be processed more efficiently.

[0229] The embodiments of the present invention have been described above with reference to the drawings (FIGS. 1A to 16). However, the present invention is not limited to the above embodiments and can be implemented in various forms without departing from the spirit of the present invention. Furthermore, the components disclosed in the above embodiments can be modified as appropriate. For example, some of the components shown in one embodiment may be added to the components of another embodiment, or some of the components shown in one embodiment may be deleted from the embodiment.

[0230] The drawings mainly show each component in a schematic manner to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown in the drawings may differ from the actual ones due to the convenience of creating the drawings. Furthermore, the configuration of each component shown in the above embodiment is merely an example and is not particularly limited, and it goes without saying that various modifications are possible within a range that does not substantially deviate from the effects of the present invention.

[0231] 1A to 16, the interrupt schedule is a time schedule for completing processing for a priority lot and processing for a regular lot, but the interrupt schedule may also indicate a time schedule for completing processing for a priority lot. In this case, the fourth remuneration condition may be omitted.

[0232] Furthermore, in the embodiment described with reference to Figures 1A to 16, the first schedule creation program CP1 was created based on the first generation program GP1, and the second schedule creation program CP2 was created based on the second generation program GP2, but it is also possible to create a single schedule creation program CP that integrates the first schedule creation program CP1 and the second schedule creation program CP2.

[0233] 1A to 16, the processed normal substrate W1a was accommodated in a substrate accommodation container CA (third substrate accommodation container CA3) different from the original substrate accommodation container CA (first substrate accommodation container CA1), but the processed normal substrate W1a may be returned to the original substrate accommodation container CA (first substrate accommodation container CA1). Similarly, the processed priority substrate W2a may be returned to the original substrate accommodation container CA (second substrate accommodation container CA2). In this case, other substrate accommodation containers CA accommodating unprocessed substrates W may be placed on the third load port LP3 and the fourth load port LP4.

[0234] 1A to 16, the container platform part LPS includes four load ports LP, but the number of load ports LP is not limited to four. The container platform part LPS may include two, three, five or more load ports LP.

[0235] Furthermore, the substrate processing apparatus 200 is not particularly limited as long as it is an apparatus for processing the substrate W. For example, the substrate processing apparatus 200 may be a chemical cleaning apparatus, a brush cleaning apparatus, a wet etching apparatus, a dry etching apparatus, a coating apparatus, a developing apparatus, an exposure apparatus, a coater developer, a baking apparatus, or a film forming apparatus.

[0236] The present invention is useful in methods and apparatus for processing substrates.

[0237] This application claims priority to Japanese Patent Application No. 2023-218109, filed December 25, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A schedule creation program generation method for generating a schedule creation program for creating a time schedule for processing a plurality of substrates by a substrate processing apparatus by reinforcement learning, wherein the substrate processing apparatus includes: a container placement unit on which a plurality of substrate storage containers for storing one or more substrates constituting one lot are placed; a plurality of substrate processing units that perform substrate processing on each substrate; and a transfer unit that transfers the substrate between the plurality of substrate storage containers and the plurality of substrate processing units. The substrate processing apparatus completes the processing of the lot by performing the substrate processing on each of the substrates constituting the lot. The schedule creation program creates an interrupt schedule in response to a second substrate storage container that accommodates a priority lot with a priority higher than the normal priority being placed on the container placement unit when all or part of the first substrates constituting a normal lot with a normal priority remain in the first substrate storage container placed in front of the container placement unit. The interrupt schedule indicates a time schedule for processing the first substrates remaining in the first substrate storage container and the plurality of second substrates constituting the priority lot by the substrate processing apparatus. The schedule creation program generation method includes a step of repeatedly performing an experience step including a time table acquisition step, an arrangement step, and a reward determination step by the reinforcement learning to increase the cumulative reward. The time table acquisition step indicates a step of acquiring a time table for defining the interrupt schedule. The arrangement step indicates a step of sequentially arranging a plurality of planned elements given to each of the first substrates remaining in the first substrate storage container and a plurality of planned elements given to each of the plurality of second substrates in the time table to sequentially change the state of the time table. The reward determination step indicates a step of determining a reward to be given to the final state of the time table based on the final state of the time table in which all the planned elements are arranged and a condition for giving a reward. The final state of the time table corresponds to the interrupt schedule. The cumulative reward indicates the total value of the rewards given to the final state of the time table.The conditions for granting the reward include a first reward condition for granting a first reward when the final state of the time table indicates that the processing for at least one of the first substrates remaining in the first substrate storage container has been completed before the processing for the priority lot is completed, and a second reward condition for granting a second reward with a value of the reward smaller than that of the first reward when the final state of the time table indicates that the processing for the priority lot has been completed before the processing for the normal lot is completed. A method for generating a scheduling program.

2. The time table acquisition step includes a step of acquiring a time table in which the planned elements given to each of the first substrates remaining inside the substrate processing apparatus are arranged, when the first substrate remains inside the substrate processing apparatus. The schedule creation program creates the interrupt schedule without changing the time schedule of the first substrate remaining inside the substrate processing apparatus. The method for generating a schedule creation program according to claim 1.

3. When all of the first substrates constituting the normal lot remain in the first substrate storage container, the first reward condition is that when the final state of the time table indicates that the processing for the number of first substrates set based on the number of the second substrates constituting the priority lot has been completed before the completion of the processing for the priority lot, the first reward is given. The method for generating a schedule creation program according to claim 1.

4. The substrate processing apparatus is capable of executing the substrate processing in parallel for two or more of the substrates. The first reward condition indicates that the first reward is given when the final state of the time table indicates that the substrate processing for the first substrate is being executed in parallel with the substrate processing for at least one of the second substrates. The method for generating a schedule creation program according to any one of claims 1 to 3.

5. The condition for giving the reward further includes a third reward condition for giving a third reward in which the earlier the time when the processing for the priority lot is completed, the larger the value of the reward. The maximum value of the third reward is smaller than the value of the first reward. The method for generating a schedule creation program according to any one of claims 1 to 3.

6. The condition for giving the reward further includes a fourth reward condition for giving a fourth reward in which the earlier the time when the processing for the normal lot is completed, the larger the value of the reward. The maximum value of the fourth reward is smaller than the value of the first reward. The method for generating a schedule creation program according to any one of claims 1 to 3.

7. The substrate processing apparatus is capable of executing the substrate processing in parallel for two or more of the substrates, and the condition for granting the reward further includes a fifth reward condition for granting a fifth reward when the final state of the time table indicates a state in which the substrate processing for the first substrate is being executed in parallel with the substrate processing for at least one of the second substrates. The schedule creation program generation method according to any one of claims 1 to 3.

8. A schedule creation program generation device that generates a schedule creation program for creating a time schedule for processing a plurality of substrates by a substrate processing apparatus by reinforcement learning, a storage unit that stores a generation program that defines the schedule creation program generation method according to any one of claims 1 to 3, and a processing unit that executes the generation program to generate the schedule creation program. A schedule creation program generation device comprising:

9. A schedule creation device for creating a time schedule for processing a plurality of substrates by a substrate processing apparatus, a storage unit that stores a schedule creation program generated based on the schedule creation program generation method according to any one of claims 1 to 3, and a processing unit that executes the schedule creation program to create the interrupt schedule. A schedule creation device comprising:

10. A computer-readable recording medium that records a generation program that defines the schedule creation program generation method according to any one of claims 1 to 3.

11. A computer-readable recording medium that records a schedule creation program generated based on the schedule creation program generation method according to any one of claims 1 to 3.

12. A generation program that is executable by a computer and that defines the schedule creation program generation method according to any one of claims 1 to 3.

13. A schedule creation program that is executable by a computer and that is generated based on the schedule creation program generation method according to any one of claims 1 to 3.

14. A substrate processing apparatus that completes processing on a lot by performing substrate processing one by one on one or more substrates constituting the lot, the apparatus comprising: a container placement unit on which a plurality of substrate storage containers for storing the substrates constituting the lot are placed; a plurality of substrate processing units that perform the substrate processing on a per-substrate basis; a transfer unit that transfers the substrate between the plurality of substrate storage containers and the plurality of substrate processing units; a storage unit that stores a generation program that defines the schedule creation program generation method according to any one of claims 1 to 3; a processing unit that executes the generation program to generate a schedule creation program for creating a time schedule for processing a plurality of substrates by the reinforcement learning, wherein when all or part of a first substrate constituting a normal lot with normal priority remains in a first substrate storage container placed in front of the container placement unit, the processing unit executes the schedule creation program in response to a second substrate storage container that stores a priority lot with a priority higher than the normal priority being placed on the container placement unit, and creates the interrupt schedule for processing the first substrate remaining in the first substrate storage container and a plurality of second substrates constituting the priority lot.

15. A substrate processing apparatus that completes processing on a lot by performing substrate processing one by one on one or more substrates constituting the lot, the apparatus comprising: a container placement unit on which a plurality of substrate storage containers for storing the substrates constituting the lot are placed; a plurality of substrate processing units that perform the substrate processing on a per-substrate basis; a transfer unit that transfers the substrate between the plurality of substrate storage containers and the plurality of substrate processing units; a storage unit that stores a schedule creation program generated based on the schedule creation program generation method according to any one of claims 1 to 3; and a processing unit that, when all or part of a first substrate constituting a normal lot with normal priority remains in a first substrate storage container placed in front of the container placement unit and a second substrate storage container that houses a priority lot with a priority higher than the normal priority is placed on the container placement unit, executes the schedule creation program to create the interrupt schedule for processing the first substrate remaining in the first substrate storage container and a plurality of second substrates constituting the priority lot.

16. A substrate processing apparatus that completes the processing for a lot by performing substrate processing one by one on one or more substrates constituting the lot, and a schedule creation program generation apparatus according to claim 8, wherein the schedule creation program generation apparatus further includes a transmission unit that transmits the schedule creation program to the substrate processing apparatus, and the substrate processing apparatus includes a container placement unit on which a plurality of substrate storage containers for storing the substrates constituting the lot are placed, a plurality of substrate processing units that perform the substrate processing on a per-substrate basis, a transfer unit that transfers the substrates between the plurality of substrate storage containers and the plurality of substrate processing units, a reception unit that receives the schedule creation program transmitted from the transmission unit of the schedule creation program generation apparatus, and a processing unit that creates the interrupt schedule for processing the first substrate remaining in the first substrate storage container and the plurality of second substrates constituting the priority lot by executing the schedule creation program in response to the placement of a second substrate storage container that houses a priority lot with a priority higher than the normal on the container placement unit when all or part of the first substrate of the normal lot with a normal priority remaining in the first substrate storage container placed prior to the container placement unit. A substrate processing system.

17. A substrate processing apparatus that completes the processing for a lot by performing substrate processing one by one on one or more substrates constituting the lot, and a schedule creation apparatus according to claim 9, wherein the schedule creation apparatus further includes a transmission unit that transmits the interrupt schedule to the substrate processing apparatus, and the substrate processing apparatus includes a container placement unit on which a plurality of substrate storage containers for storing the substrates constituting the lot are placed, a plurality of substrate processing units that perform the substrate processing on a per-substrate basis, a transfer unit that transfers the substrates between the plurality of substrate storage containers and the plurality of substrate processing units, and a reception unit that receives the interrupt schedule transmitted from the transmission unit of the schedule creation apparatus. A substrate processing system.

Images