Computer program, data generation method, learning model generation method, and information processing device

Abstract

To provide a computer program, a data generation method, a learning model generation method, and an information processing device by which it can be expected that an operation analysis of image-based substrate processing and the like are assisted.SOLUTION: A computer program according to the present embodiment causes a computer to execute processing of: acquiring a moving image obtained by imaging substrate processing; and generating extension data based on a frame image at a first point of time included in the acquired moving image and a frame image at a second point of time later than the first point of time.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a computer program, a data generation method, a learning model generation method, and an information processing device. [Background technology]

[0002] Patent Document 1 proposes a substrate processing method including a holding step of transporting and holding a substrate inside a chamber, a supply step of supplying a fluid to the substrate inside the chamber, an imaging step of using a camera to sequentially capture images of the inside of the chamber to obtain image data, a condition setting step of identifying a monitoring target from a plurality of candidate monitoring targets inside the chamber and changing image conditions based on the monitoring target, and a monitoring step of performing monitoring processing on the monitoring target based on image data having image conditions according to the monitoring target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-190511 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a computer program, a data generation method, a learning model generation method, and an information processing device that are expected to support image-based analysis of substrate processing operations, etc. [Means for solving the problem]

[0005] A computer program according to one embodiment causes a computer to acquire video images of substrate processing and generate extended data based on a frame image at a first point in time contained in the acquired video images and a frame image at a second point in time that is later than the first point in time. [Effects of the Invention]

[0006] According to the present disclosure, it is expected that image-based analysis of substrate processing operations and the like can be supported. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram for explaining a configuration example of a substrate processing apparatus according to an embodiment of the present invention; [Figure 2] 1 is a schematic diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Figure 3] 1 is a block diagram showing an example of a configuration of an information processing device according to an embodiment of the present invention; [Figure 4] 10 is a schematic diagram for explaining data extension performed by the information processing device according to the present embodiment. FIG. [Figure 5] FIG. 2 is a schematic diagram showing an example of a label information input screen displayed by the information processing device according to the present embodiment. [Figure 6] 10 is a flowchart illustrating an example of a procedure of a learning data generation process performed by the information processing device according to the present embodiment. [Figure 7] FIG. 2 is a schematic diagram illustrating an example of the configuration of a learning model generated by an information processing device. [Figure 8] 10 is a flowchart illustrating an example of a procedure for a learning model generation process performed by the information processing device in the present embodiment. [Figure 9] FIG. 10 is a schematic diagram illustrating an example of the configuration of a learning model generated by an information processing device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Specific examples of information processing systems according to embodiments of the present disclosure will be described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0009] [Embodiment 1] <System configuration> 1 is a schematic diagram illustrating an example of the configuration of a substrate processing apparatus 1 according to this embodiment. The substrate processing apparatus 1 according to this embodiment is an apparatus that performs substrate processing, known as wet etching, in which a substrate to be processed (e.g., a wafer having an oxide film, nitride film, or the like formed thereon) is rotated while a chemical solution that dissolves the film is supplied onto the film to process the substrate into a desired shape. The substrate processing apparatus 1 according to this embodiment is configured to include a chamber 11, a substrate holding mechanism 12, a discharge unit 13, and a collection cup 14.

[0010] The chamber 11 is a sealed reaction vessel, and houses therein a substrate holding mechanism 12, a discharge unit 13, a collection cup 14, etc. An FFU (Fan Filter Unit) 15 is provided on the ceiling of the chamber 11. The FFU 15 forms a downflow within the chamber 11.

[0011] The substrate holding mechanism 12 has a holder 12a, a support 12b, and a drive unit 12c. The holder 12a is, for example, disk-shaped and holds a substrate (wafer) to be processed horizontally on the disk. The support 12b is a cylindrical member connected to the center of the underside of the holder 12a and extending vertically (up and down in FIG. 1 ), supporting the holder 12a horizontally. The lower end of the support 12b is connected to the drive unit 12c and is rotatably supported by the drive unit 12c. The drive unit 12c has a prime mover such as a motor and rotates the support 12b around its axis. As a result, the substrate holding mechanism 12 rotates the support 12b using the drive unit 12c, thereby rotating the holder 12a supported by the support 12b and rotating the substrate held by the holder 12a.

[0012] The discharge unit 13 discharges a liquid such as a chemical solution or a cleaning solution onto the substrate held by the substrate holding mechanism 12. For example, dilute hydrofluoric acid is used as the chemical solution, and pure water is used as the cleaning solution, but the liquids discharged by the discharge unit 13 are not limited to these. The discharge unit 13 is connected to a liquid supply source 16 provided outside the chamber 11 via, for example, a tubular liquid supply path, and discharges the liquid supplied from this supply source 16 onto the substrate. The discharge unit 13 is also connected to a drive mechanism (not shown) and can move horizontally between the center and peripheral edges of the substrate. By combining the rotation of the substrate by the substrate holding mechanism 12 and the horizontal movement of the discharge unit 13 by the drive mechanism, the substrate processing apparatus 1 can discharge the liquid from the discharge unit 13 to an appropriate position on the substrate to be processed.

[0013] Recovery cup 14 is disposed so as to surround holding portion 12a of substrate holding mechanism 12, and collects liquid scattered from the substrate due to the rotation of holding portion 12a. A drainage port 14a is provided at the bottom of recovery cup 14, and liquid collected by recovery cup 14 is discharged from drainage port 14a to the outside of chamber 11. In addition, an exhaust port 14b is provided at the bottom of recovery cup 14, and gas supplied from FFU 15 is discharged from exhaust port 14b to the outside of chamber 11.

[0014] 1 is configured to include one discharge unit 13 that discharges a liquid. The substrate processing apparatus 1 can selectively discharge either a chemical liquid for dissolving a substrate or a cleaning liquid for cleaning the substrate by switching between the chemical liquid for dissolving the substrate and the cleaning liquid using a supply source 16. However, the substrate processing apparatus 1 may be configured to include multiple discharge units 13. For example, the substrate processing apparatus 1 may be configured to include a separate discharge unit 13 for discharging a chemical liquid and a separate discharge unit 13 for discharging a cleaning liquid.

[0015] FIG. 2 is a schematic diagram for explaining an overview of an information processing system according to this embodiment. The information processing system according to this embodiment is configured to include the above-described substrate processing apparatus 1, an information processing apparatus 3, and a camera 5. The camera 5 has an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and can capture so-called moving images by capturing images several tens of times per second. The camera 5 is provided, for example, in the chamber 11 of the substrate processing apparatus 1, and captures images of the discharge unit 13 during substrate processing. The camera 5 transmits data of the moving images obtained by capturing the images to the information processing apparatus 3. The moving image data is, for example, data in which a plurality of still images (frame images) are arranged in time series. The camera 5 may be, for example, a device provided in the substrate processing apparatus 1, or may be, for example, a device separate from the substrate processing apparatus 1.

[0016] The information processing device 3 is a device that performs processes of generating learning data for machine learning based on data of moving images captured by the camera 5 and generating a learning model by machine learning using the learning data. In the present embodiment, the information processing device 3 is provided as a device separate from the substrate processing device 1, but this is not limiting and the information processing device 3 may be a device integrated with the substrate processing device 1. The information processing device 3 is connected to the camera 5 via, for example, a communication cable, and can send and receive data to and from the camera 5. The information processing device 3 receives data of moving images of the discharge portion 13 transmitted by the camera 5, and stores and accumulates the received data of moving images in a storage unit.

[0017] The information processing device 3 according to this embodiment generates learning data for generating a learning model that determines the state of liquid discharge by the discharge unit 13, based on frame images included in moving images acquired and accumulated from the camera 5. At this time, the information processing device 3 according to this embodiment increases the number of frame images used for machine learning by performing so-called data expansion, in which frame images not included in the moving images captured by the camera 5 are generated based on the frame images included in the moving images. For example, the information processing device 3 receives input from the user of label information that indicates the discharge state of the discharge unit 13 for each frame image, and uses a set of data in which each frame image is associated with the label information as learning data for so-called supervised machine learning.

[0018] When the information processing device 3 generates learning data for so-called unsupervised machine learning, it is not necessary to accept input of label information or associate label information with each frame image. In this case, the information processing device 3 uses as learning data a data set including frame images included in a moving image captured by the camera 5 and frame images generated by extending these frame images.

[0019] The information processing device 3 also performs machine learning processing using the generated learning data to generate a learning model. For example, if the learning data is a set of frame images associated with label information, the information processing device 3 can perform supervised machine learning to receive the frame images as input and generate a learning model that classifies the discharge state of the discharge unit 13 of the substrate processing device 1 captured in the frame images. The learning model generated by the information processing device 3 is installed in, for example, a control device that controls the operation of the substrate processing device 1, and the control device acquires video images of the discharge unit 13 of the substrate processing device 1 captured by the camera 5. The control device inputs the frame images included in the acquired video images to the learning model, acquires the classification result of the discharge state output by the learning model, and can perform control processing such as stopping substrate processing by the substrate processing device 1 if the acquired classification result indicates, for example, the occurrence of an abnormality.

[0020] 3 is a block diagram showing an example of the configuration of an information processing device 3 according to this embodiment. The information processing device 3 according to this embodiment can be realized by installing a predetermined application program or the like in a general-purpose information processing device such as a personal computer or a server computer. The information processing device 3 according to this embodiment is configured to include a processing unit 31, a storage unit 32, a communication unit 33, a display unit 34, and an operation unit 35. Note that, in this embodiment, the processing will be described as being performed by one information processing device 3, but the processing of the information processing device 3 may be distributed among a plurality of devices.

[0021] The processing unit 31 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processing unit 31 reads and executes a program 32a stored in the storage unit 32 to perform various processes, such as generating learning data for machine learning based on moving images acquired from the camera 5, and generating a learning model by machine learning using the generated learning data.

[0022] The storage unit 32 is configured using a large-capacity storage device such as a hard disk or an SSD (Solid State Drive). The storage unit 32 stores various programs executed by the processing unit 31 and various data required for the processing of the processing unit 31. In this embodiment, the storage unit 32 stores a program 32a executed by the processing unit 31. The storage unit 32 also includes a learning data storage unit 32b that stores generated learning data, and a model information storage unit 32c that stores information related to the generated learning model.

[0023] In this embodiment, the program (computer program, program product) 32a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disc, and the information processing device 3 reads the program 32a from the recording medium 99 and stores it in the storage unit 32. However, the program 32a may also be written to the storage unit 32, for example, during the manufacturing stage of the information processing device 3. Furthermore, for example, the program 32a may be distributed by a remote server device or the like and acquired by the information processing device 3 via communication. For example, the program 32a may be recorded on the recording medium 99 and read by a writing device and written to the storage unit 32 of the information processing device 3. The program 32a may be provided in a form distributed via a network, or may be provided in a form recorded on the recording medium 99.

[0024] The learning data storage unit 32b stores learning data generated by the information processing device 3 based on moving images captured by the camera 5. The learning data is, for example, data in which frame images are associated with label information indicating the discharge state of the discharge unit 13. The model information storage unit 32c stores information about a learning model that has undergone machine learning. The information about the learning model may include, for example, information indicating the configuration of the learning model and information such as the values ​​of internal parameters determined by machine learning.

[0025] In this embodiment, the generation of learning data and the generation of a learning model are both performed by the information processing device 3, but this is not limited to this. The generation of learning data and the generation of a learning model may be performed by different devices. The device that generates the learning data transmits the learning data to the device that generates the learning model, and the device that generates the learning model receives the learning data and performs machine learning processing.

[0026] The communication unit 33 transmits and receives data to and from the camera 5, for example, via a wired or wireless network N. In this embodiment, the communication unit 33 receives moving image data transmitted from the camera 5 and provides the data to the processing unit 31. The communication unit 33 may also transmit to the camera 5, for example, a command to control the operation of the camera 5, based on information provided by the processing unit 31.

[0027] The display unit 34 is configured using a liquid crystal display or the like, and displays various images, characters, and the like based on the processing of the processing unit 31. The display unit 34 displays various information, such as images (moving images or frame images) captured by the camera 5, a screen for receiving input of label information relating to the discharge state of the discharge unit 13 from the user, or the progress status of machine learning that generates a learning model.

[0028] The operation unit 35 accepts user operations and notifies the processing unit 31 of the accepted operations. For example, the operation unit 35 accepts user operations using an input device such as a mechanical button or a touch panel provided on the surface of the display unit 34. Furthermore, for example, the operation unit 35 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 3.

[0029] The storage unit 32 may be an external storage device connected to the information processing device 3. The information processing device 3 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The information processing device 3 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, and may not include, for example, the display unit 34 and the operation unit 35.

[0030] In the information processing device 3 according to this embodiment, the processing unit 31 reads and executes the program 32a stored in the storage unit 32, whereby an image acquisition unit 31a, a data expansion unit 31b, a learning data generation unit 31c, a learning processing unit 31d, a display processing unit 31e, and the like are realized as software functional units in the processing unit 31. Note that in the figure, functional units that perform processing related to the generation of learning data and the generation of a learning model are shown as functional units of the processing unit 31, and functional units related to processing other than these are not shown.

[0031] The image acquisition unit 31a communicates with the camera 5 via the communication unit 33, thereby acquiring image data of the discharge unit 13 of the substrate processing apparatus 1 captured by the camera 5. In this embodiment, the camera 5 is a camera that captures moving images by capturing images several tens of times per second. The image data acquired by the image acquisition unit 31a may be in the form of moving images, or may be in the form of still images (frame images) included in the moving images. The image acquisition unit 31a repeatedly acquires images from the camera 5 and stores the images acquired from the camera 5 in the learning data storage unit 32b. The image acquisition unit 31a also repeatedly acquires images from the camera 5 while the substrate processing is being performed, thereby enabling it to obtain time-series frame images of the discharge unit 13.

[0032] The data expansion unit 31b performs a process of increasing the number of frame images by performing data expansion based on the video (multiple frame images in time series) captured by the camera 5. In this embodiment, the data expansion unit 31b performs data expansion by extracting a frame image at a certain point in time and a frame image at the next point in time from the multiple frame images in time series contained in the video, and generating frame images corresponding to points in time between these two points in time. The data expansion unit 31b stores the generated frame images in the learning data storage unit 32b.

[0033] The learning data generation unit 31c performs a process of generating learning data for machine learning based on frame images included in the video acquired by the image acquisition unit 31a and frame images generated by the data expansion unit 31b through data expansion. For example, the learning data generation unit 31c performs a process of accepting, from a user, input of label information to be included in the learning data for supervised machine learning. For example, the learning data generation unit 31c displays frame images on the display unit 34 and accepts input of label information indicating the ejection state of the ejection unit 13 depicted in the displayed frame images based on a user's operation on the operation unit 35. The learning data generation unit 31c associates the displayed frame images with the label information input by the user and stores them in the learning data storage unit 32b. The learning data generation unit 31c accepts input of label information for multiple frame images, including frame images included in the video acquired by the image acquisition unit 31a and frame images generated by the data expansion unit 31b through data expansion, and creates a dataset containing multiple pairs of frame images and label information as learning data. When generating learning data for unsupervised learning that does not include label information, the learning data generation unit 31c does not accept input of label information, but simply collects frame images included in the video acquired by the image acquisition unit 31a and frame images generated by the data expansion unit 31b through data expansion into a single data set, which can be used as learning data.

[0034] The learning processing unit 31d performs machine learning processing using the learning data stored in the learning data storage unit 32b to generate a learning model that makes predictions based on frame images. The learning processing unit 31d performs supervised machine learning using learning data in which, for example, frame images and label information indicating the ejection state are associated with each other to generate a learning model that accepts frame images as input and outputs information indicating the ejection state of the ejection unit 13 depicted in the frame image. The learning model may be configured, for example, using a convolutional neural network (CNN) or a deep neural network (DNN), but is not limited to these and may have any configuration. The learning processing unit 31d performs machine learning processing using an existing method, for example, a stochastic gradient descent method or an error backpropagation method. The configuration of the learning model and the machine learning method for generating the learning model are existing technologies, so detailed description thereof will be omitted in this embodiment.

[0035] The display processing unit 31e performs processing to display various characters, images, and the like on the display unit 34. In the present embodiment, the display processing unit 31e displays, for example, a moving image acquired by the image acquisition unit 31a or a frame image included in the moving image on the display unit 34. The display processing unit 31e also displays, for example, a plurality of selection items indicating the ejection state on the display unit 34 to accept input of the ejection state of the ejection unit 13 depicted in the displayed frame image. The user performs an operation to select one of the selection items displayed on the display unit 34 that is appropriate as the ejection state of the ejection unit 13 depicted in the displayed frame image, and the information processing device 3 accepts this operation via the operation unit 35, thereby accepting input of label information for the ejection state for the frame image. The display processing unit 31e also displays, for example, information such as the number of learning iterations or the evaluation value of the learning model as the progress of the machine learning process on the display unit 34.

[0036] <Learning data generation process> The information processing device 3 according to this embodiment performs processing to generate learning data for machine learning that generates a learning model, based on frame images included in a moving image of the discharge unit 13 of the substrate processing device 1 captured by the camera 5. In this case, the information processing device 3 can increase the amount of learning data by performing data expansion processing based on multiple frame images included in the moving image.

[0037] 4 is a schematic diagram for explaining data extension performed by information processing device 3 according to this embodiment. In this diagram, frame images included in a moving image captured by camera 5 are shown in chronological order, with the names "frame + integer value" attached, such as frame 1, frame 2, and frame 3. Also in this diagram, new frame images generated by information processing device 3 through data extension of these frame images are shown with the names "extended frame + decimal value" attached, such as extended frame 1.9 and extended frame 2.5.

[0038] The information processing device 3 extracts two frame images: a frame image at a certain point in time included in a moving image, and the next frame image in chronological order of the frame image. The information processing device 3 calculates the average value of corresponding pixels in the two extracted frame images, and generates a frame image whose pixel value is the calculated average value as an extended frame image. In the example shown in the figure, extended frame 2.5 is generated based on the average value of frames 2 and 3.

[0039] Furthermore, the information processing device 3 may generate an extended frame image by calculating a weighted average value rather than a simple average value. In the example shown in the figure, extended frame 1.9 is generated based on the average value calculated by weighting frame 1 and frame 2 at a ratio of 1:9. The information processing device 3 can generate an extended frame image by, for example, randomly weighting two frame images extracted from a video and calculating a weighted average value. Furthermore, when generating an extended frame image from two frame images, the information processing device 3 can employ various methods, such as linear interpolation or cubic interpolation, rather than calculating an average value or weighted average value.

[0040] The information processing device 3 stores, as a dataset, frame images included in the original video and extended frame images generated by extending these frame images in the learning data storage unit 32b. This dataset can be used as learning data for performing unsupervised machine learning to generate a learning model such as an autoencoder. The information processing device 3 according to this embodiment also generates a dataset in which label information that indicates the correct value for prediction is attached to each frame image (original frame image and extended frame image), as learning data for generating a learning model that performs prediction such as classification or regression based on images using supervised machine learning. The information processing device 3 acquires the label information to be attached to each frame image through user input.

[0041] 5 is a schematic diagram showing an example of a label information input screen displayed by the information processing device 3 according to this embodiment. In this example, it is assumed that the user performs a so-called annotation task, in which the user selects either a label of "droplets present" to indicate a state in which droplets are falling from the discharge unit 13 onto the substrate to be processed, or a label of "no droplets" to indicate a state in which droplets are not falling, for a frame image capturing the discharge unit 13 of the substrate processing device 1. However, this is just one example, and any label information may be attached to the frame image.

[0042] In this embodiment, the information processing device 3 appropriately extracts one frame image from, for example, multiple frame images (original frame image and extended frame image) and displays it in the left region of the label information input screen. In the right region of the same screen, a message string saying "Please select the ejection state" and buttons labeled "With droplets" and "Without droplets" are displayed vertically. A user can input label information for a frame image by clicking or touching either the "With droplets" button or the "Without droplets" button using an operation unit 35 such as a mouse or a touch panel. The information processing device 3 accepts the user's selection of label information in response to an operation on one of the buttons, and stores the selected label information in the learning data storage unit 32b in association with the frame image displayed on the label information input screen. The information processing device 3 accepts the user's input of label information for multiple frame images in sequence and stores the accepted label information in the learning data storage unit 32b, thereby creating a dataset of sets of frame images and label information as learning data.

[0043] 6 is a flowchart showing an example of the procedure of the learning data generation process performed by the information processing device 3 according to this embodiment. In this embodiment, the image acquisition unit 31a of the processing unit 31 of the information processing device 3 acquires moving image data captured by the camera 5 while substrate processing is being performed in the substrate processing device 1 by communicating with the camera 5 via the communication unit 33 (step S1). The image acquisition unit 31a stores the moving image data (frame images included in the moving image data) acquired in step S1 in the learning data storage unit 32b of the storage unit 32 (step S2). The image acquisition unit 31a determines whether the substrate processing by the substrate processing device 1 has been completed (step S3). If the substrate processing has not been completed (S3: NO), the image acquisition unit 31a returns to step S1 and repeatedly acquires and stores moving images until the substrate processing is completed. Whether the substrate processing is completed may be determined, for example, by the information processing device 3 communicating with the substrate processing device 1, or may be determined based on moving images captured by the camera 5.

[0044] If the substrate processing is completed (S3: YES), the data expansion unit 31b of the processing unit 31 acquires a frame image (e.g., the first frame image) at a certain time point from among the multiple frame images included in the moving image data stored in step S2 (step S4). The data expansion unit 31b also acquires a frame image at a time point next in time in the chronological order of the frame image acquired in step S4 (step S5). The data expansion unit 31b generates a frame image intermediate between the certain time point and the next time point by, for example, calculating the average value of corresponding pixels for the frame image at the certain time point and the frame image at the next time point (step S6). The data expansion unit 31b stores the expanded frame image generated in step S6 in the learning data storage unit 32b (step S7). The data expansion unit 31b determines whether data expansion has been completed for all frame images acquired and stored in steps S1 and S2 (step S8). If data extension has not been completed for all frame images (S8: NO), the data extension unit 31b returns the process to step S5, acquires a frame image at the next point in time, and repeats the same process.

[0045] If data extension has been completed for all frame images (S8: YES), the learning data generation unit 31c of the processing unit 31 displays, for example, the label information input screen shown in Fig. 5 on the display unit 34 and accepts input of label information for all frame images included in the video and frame images generated by data extension (step S9). The learning data generation unit 31c associates the label information accepted in step S9 with each frame image and stores it as learning data in the learning data storage unit 32b (step S10), and ends the learning data generation process.

[0046] In this embodiment, the information processing device 3 uses both the frame images included in the video and the extended frame images generated by extending these frame images as learning data, but this is not limited to this. The information processing device 3 may also use only the extended frame images generated by data extension processing of the original frame images as learning data.

[0047] <Learning model generation process> The information processing device 3 according to this embodiment performs processing to generate a learning model that makes predictions regarding substrate processing using learning data generated based on video images captured by the camera 5. FIG. 7 is a schematic diagram showing an example of the configuration of a learning model generated by the information processing device 3. In this example, the learning model 101 generated by the information processing device 3 is a learning model that receives as input an image of the discharge unit 13 of the substrate processing device 1 and classifies the discharge state of the discharge unit 13 as either "droplets present" or "no droplets present." This learning model 101 may be configured, for example, as a CNN. In response to the input of the image of the discharge unit 13, the learning model 101 outputs two numerical values: one indicating the possibility that the discharge state of the discharge unit 13 is "droplets present" and the other indicating the possibility that the discharge state is "no droplets present." The larger numerical value is the classification result.

[0048] The information processing device 3 can generate the illustrated learning model 101 by performing supervised machine learning using the learning data generated in the above-described learning data generation process, for example, learning data in which frame images of the discharge unit 13 of the substrate processing device 1 are associated with label information of "droplets present" or "droplets absent." In supervised machine learning, the information processing device 3 inputs the frame images of the learning data to the learning model 101, and generates the learning model 101 by updating the internal parameters of the learning model 101 in response to this input so that the information output by the learning model 101 approaches the label information of the learning data.

[0049] Fig. 8 is a flowchart showing an example of the procedure of the learning model generation process performed by the information processing device 3 in this embodiment. Note that this figure shows the procedure when the learning model 101 shown in Fig. 7 is generated by performing supervised machine learning. In this embodiment, the learning processing unit 31d of the processing unit 31 of the information processing device 3 sets initial values ​​of internal parameters for the learning model 101 having a predetermined structure such as CNN (step S21). The initial values ​​of the internal parameters may be, for example, predetermined values, or may be determined randomly, or may be values ​​of internal parameters of the learning model 101 that has previously undergone some learning.

[0050] The learning processing unit 31d acquires one piece of learning data stored in the learning data storage unit 32b (step S22). The learning processing unit 31d inputs a frame image included in the learning data acquired in step S22 to the learning model 101 (step S23). The learning processing unit 31d acquires a classification result of "droplets present" or "droplets absent" output by the learning model 101 in response to the input in step S23 (step S24). The learning processing unit 31d calculates an error in the classification result acquired in step S24 based on the label information included in the learning data acquired in step S22 (step S25). The learning processing unit 31d updates the internal parameters of the learning model 101 based on the error calculated in step S25, for example, by backpropagation (step S26).

[0051] The learning processing unit 31d determines whether a condition for terminating the machine learning has been met, such as the number of repetitions exceeding a threshold or the prediction accuracy of the learning model 101 reaching a target value (step S27). If the termination condition has not been met (S27: NO), the learning processing unit 31d returns to step S22 and repeats the above-mentioned process. If the termination condition has been met (S27: YES), the learning processing unit 31d stores the internal parameters of the learning model 101 in the model information storage unit 32c (step S28) and terminates the learning model generation process.

[0052] In this embodiment, the information output by the learning model 101 is two types of ejection states, "droplets present" or "droplets absent," but this is not limited to this. The learning model 101 may be configured to output information indicating three or more types of ejection states.

[0053] For example, the learning model 101 can be configured to output information indicating four discharge states: "liquid column present," "liquid column broken and falling," "droplets present," and "no liquid." The "liquid column present" discharge state is a state in which liquid is continuously discharged from the discharge unit 13, and a columnar liquid (liquid column) connects the lower end of the discharge unit 13 and the upper surface of the substrate. The "liquid column broken and falling" discharge state is a state immediately after discharge of liquid from the discharge unit 13 has stopped, in which a space exists between the lower end of the discharge unit 13 and the upper end of the liquid column, and a liquid column stands on the upper surface of the substrate. The "droplets present" discharge state is a state in which no liquid column exists between the lower end of the discharge unit 13 and the upper surface of the substrate, and one or more spherical liquids (droplets) exist. The "no liquid" discharge state is a state in which neither a liquid column nor droplets exist between the lower end of the discharge unit 13 and the upper surface of the substrate.

[0054] Furthermore, the learning model 101 may be a regression model that predicts some numerical value related to the discharge, rather than a classification model that classifies the discharge state. For example, the learning model 101 may be configured to predict and output a numerical value such as the size or amount of droplets captured in a frame image.

[0055] <Summary> In the information processing system according to this embodiment having the above configuration, the information processing device 3 acquires video images of substrate processing by the substrate processing device 1 captured by the camera 5, and generates extended data (extended frame images) based on frame images at a first time point included in the acquired video images and frame images at a second time point that follows. As a result, in the information processing system according to this embodiment, the number of frame images used for machine learning and the like can be increased by using the frame images obtained from the video images captured by the camera 5 and the extended frame images generated by data expansion, and it is expected that image-based analysis of substrate processing operations and the like will be supported.

[0056] Furthermore, in the information processing system according to this embodiment, the information processing device 3 acquires moving images of the discharge unit 13 of the substrate processing apparatus 1 discharging liquid onto the substrate to be processed, and generates extended data. As a result, the information processing system according to this embodiment is expected to generate extended data for performing machine learning to generate a learning model 101 that determines the discharge state of the discharge unit 13 of the substrate processing apparatus 1.

[0057] Furthermore, in the information processing system according to this embodiment, the information processing device 3 generates extended data based on the average or weighted average of the frame image at the first time point and the frame image at the second time point, etc. This makes it possible to expect that the information processing system according to this embodiment will be able to generate extended data with simple calculations.

[0058] Furthermore, in the information processing system according to this embodiment, the information processing device 3 performs machine learning using learning data including the generated extended data to generate a learning model that outputs information about the state of substrate processing from frame images included in a video captured of the substrate processing device 1. As a result, the information processing system according to this embodiment is expected to determine the state of substrate processing using the generated learning model and realize control of the substrate processing according to the determined state.

[0059] [Embodiment 2] The learning data generated by the information processing device 3 in the above-described first embodiment is intended to generate a learning model 101 that accepts an image as an input. In contrast, the learning data generated by the information processing device 3 according to the second embodiment is learning data for generating a learning model that accepts feature data generated based on an image as an input.

[0060] FIG. 9 is a schematic diagram showing an example of the configuration of a learning model generated by the information processing device 3 according to the second embodiment. The learning model 102 according to the second embodiment is a learning model that receives, as an input, feature quantities of an image captured of the discharge unit 13 of the substrate processing device 1 and classifies the discharge state of the discharge unit 13 as either "droplets present" or "droplets absent." The information processing device 3 according to the second embodiment includes a feature quantity extraction model 103 that extracts feature quantities from frame images included in a moving image captured by the camera 5. The feature quantity extraction model 103 is a learning model that is generated by machine learning in advance in the information processing device 3 or another device, and information such as internal parameters is stored in the model information storage unit 32c. Note that the learning model that converts an image into feature quantities is an existing technology, and therefore detailed description of the configuration of the learning model, the learning method, and the like will be omitted.

[0061] The information processing device 3 according to the second embodiment generates data, as learning data for performing machine learning of the learning model 103 shown in Fig. 9, in which feature amounts extracted by the feature extraction model 103 from frame images included in a moving image captured by the camera 5 are associated with label information indicating whether the ejection state of the ejection unit 13 captured in the original frame image is "droplets present" or "droplets absent." The information processing device 3 acquires the moving image captured by the camera 5, acquires frame images included in the acquired moving image, inputs the acquired frame images to the feature extraction model 103, and acquires the feature amounts output by the feature extraction model 103. In this way, the information processing device 3 can acquire multiple feature amounts corresponding to multiple frame images included in the moving image.

[0062] Furthermore, the information processing device 3 according to the second embodiment performs data expansion on a plurality of feature quantities extracted from frame images included in a video to increase the number of feature quantities used as learning data. Data expansion on feature quantities can be performed by techniques such as averaging, weighted averaging, linear interpolation, or cubic interpolation, similar to the data expansion on frame images performed in the first embodiment. For example, when the feature extraction model 103 converts an image into an N-dimensional feature quantity vector (N is a natural number), the information processing device 3 can obtain a data-expanded N-dimensional feature quantity vector by calculating N average values ​​of two corresponding values ​​for N values ​​of feature quantities at a certain point in time and N values ​​of feature quantities at the next point in time.

[0063] The information processing device 3 according to the second embodiment can display a screen similar to the label information input screen shown in Fig. 5 on the display unit 34, accept input of label information for frame images from the user, and associate the accepted label information with features extracted from the displayed frame images to create learning data. Note that in order to accept input of label information for features generated by data augmentation, the information processing device 3 can calculate the average value of two frame images corresponding to the two features that were the basis for generating this feature, generate an image for display, and display the generated image on the label information input screen.

[0064] When generating learning data for unsupervised machine learning, the information processing device 3 can use a set of features extracted from frame images contained in a moving image and features generated by data expansion based on these features as a single data set as the learning data.

[0065] In the information processing system according to the second embodiment configured as described above, the information processing device 3 converts frame images included in a moving image into feature data, and generates extended data based on the feature data of the frame image at a first time point and the feature data of the frame image at a second time point. As a result, the information processing system according to this embodiment can use the generated extended data to generate a learning model that predicts the state of substrate processing, etc., based on the feature data, and is expected to support image-based analysis of substrate processing operations, etc.

[0066] The other configurations of the information processing system according to the second embodiment are the same as those of the information processing system according to the first embodiment, so the same reference numerals are used for the same parts and detailed description thereof will be omitted.

[0067] [Embodiment 3] In the information processing systems according to the first and second embodiments described above, when the substrate processing apparatus 1 performs wet etching as a substrate process, the camera 5 captures an image of the discharge unit 13 to obtain a moving image, and the frame images included in the obtained moving image are subjected to data expansion to generate learning data. However, the subject of data expansion and generation of learning data is not limited to a moving image captured of the discharge unit 13 during wet etching.

[0068] In substrate processing performed by the substrate processing apparatus 1, etching, ion implantation, or the like is performed using resist formed on the substrate to be processed as a mask, and then the unnecessary resist is removed from the substrate. To remove the resist, a remover liquid such as SPM (Sulfuric Acid Hydrogen Peroxide Mixture), which is a mixture of sulfuric acid and hydrogen peroxide, is used. The remover liquid is heated to a high temperature to enhance its resist removal ability and then discharged from the discharge unit 13 onto the substrate to be processed. In the information processing system according to the third embodiment, when the substrate processing apparatus 1 removes the resist formed on the substrate, the camera 5 captures an image of the discharge unit 13 discharging the remover liquid to obtain a moving image, and frame images included in the obtained moving image are data-extended to generate learning data.

[0069] The information processing device 3 according to the third embodiment performs data expansion on frame images included in a moving image captured of the discharge unit 13 when performing the resist removal process, thereby increasing the amount of generated learning data. Similarly to the information processing device 3 according to the second embodiment, the information processing device 3 according to the third embodiment may convert the frame images into feature quantities and perform data expansion on the feature quantities. Data expansion of the frame images or feature quantities may be performed by techniques such as average value, weighted average value, linear interpolation, or cubic interpolation.

[0070] During the resist removal process, a high-temperature removal liquid is discharged, generating steam inside chamber 11, and the steam may reduce visibility in the moving images captured by camera 5. The extended frame images generated by information processing device 3 based on frame images included in the moving images are generated by data extension based on average values, etc., and are therefore expected to reduce the influence of steam in the frame images. To achieve this effect, information processing device 3 according to the third embodiment may not include frame images included in the moving images captured by camera 5 in the training data, but may include only extended frame images generated by data extension based on the frame images.

[0071] The other configurations of the information processing system according to the third embodiment are the same as those of the information processing systems according to the first and second embodiments, so the same reference numerals are used for the same parts and detailed description thereof will be omitted.

[0072] Although wet etching has been described as an example of substrate processing in the first and second embodiments, and resist removal has been described as an example in the third embodiment, the generation of extended data by the information processing system according to the present embodiments is not limited to these substrate processing operations and can be applied to various other substrate processing operations. Furthermore, the location photographed by camera 5 in substrate processing apparatus 1 is not limited to discharge unit 13, and various other locations may be photographed.

[0073] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0074] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limited to this format. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used. [Explanation of symbols]

[0075] 1. Substrate processing equipment 3. Information processing equipment (computers) 5. Camera 11 Chamber 12 Board holding mechanism 12a Holding part 12b Support section 12c Drive unit 13 Discharge part 14 Collection cup 14a Drainage port 14b Exhaust port 15 FFU 16 Source 31 Processing section 31a Image acquisition section 31b Data extension 31c Learning data generation unit 31d Learning processing unit 31e Display processing section 32 Storage section 32a Program (computer program) 32b Learning data storage unit 32c Model information storage section 33 Communications Department 34 Display section 35 Control section N Network

Claims

1. Acquire video images of the substrate processing, generating extended data based on a frame image at a first time point included in the acquired video and a frame image at a second time point after the first time point; A computer program that causes a computer to perform a process.

2. acquiring a moving image of a discharge unit of the substrate processing apparatus that discharges a liquid onto a substrate to be processed; 2. The computer program of claim 1.

3. generating the extended data based on an average of the frame image at the first time point and the frame image at the second time point; 2. The computer program of claim 1.

4. converting the frame images into feature data; generating the extended data based on feature amount data obtained by converting the frame image at the first time point and feature amount data obtained by converting the frame image at the second time point; 2. The computer program of claim 1.

5. generating the extended data based on an average of the feature amount data at the first time point and the feature amount data at the second time point; 5. A computer program according to claim 4.

6. generating the augmented data based on a weighted average; 6. A computer program according to claim 3 or claim 5.

7. generating a learning model that outputs information about the state of the substrate processing from frame images included in a video of the substrate processing, by machine learning using the generated extended data; 2. The computer program of claim 1.

8. The information processing device Acquire video images of the substrate processing, generating extended data based on a frame image at a first time point included in the acquired video and a frame image at a second time point after the first time point; Data generation method.

9. The information processing device Acquire video images of the substrate processing, generating extended data based on a frame image at a first time point included in the acquired moving image and a frame image at a second time point that is later than the first time point; A learning model is generated by machine learning using the generated extended data, and the learning model outputs information about the state of the substrate processing from frame images included in a video image of the substrate processing. Learning model generation method.

10. a processing unit; The processing unit Acquire video images of the substrate processing, generating extended data based on a frame image at a first time point included in the acquired video and a frame image at a second time point after the first time point; Information processing device.

Images