Data processing system and data processing method

Abstract

In order to dynamically identify and collect attribute information to be collected in a learning data set of a machine learning model, the present invention proposes a data processing system that executes: a process for calculating third information which is an evaluation value from at least one of first information and second information; a process for calculating a statistic by performing a statistical operation on a plurality of pieces of the third information, and deciding, on the basis of the statistic, on a collection range for collecting the third information; a process for collecting data included in the collection range of the third information from among a plurality of pieces of third information calculated from the output of at least one evaluation device; and a process for re-training the learning model by using the collected third information and first information and second information which correspond to the collected third information.

Description

Data processing system and data processing method

[0001] The present invention relates to a data processing system and a data processing method.

[0002] The performance of a machine learning model depends on the quality of the training data. The quality of the training data depends on the comprehensiveness of the data attributes that can be input to the machine learning model. Therefore, it is desirable for the training data to be collected in a way that covers a wide range of variable factors that can be input to the model.

[0003] For example, Patent Literature 1 discloses a method for identifying missing attributes during learning, collecting data on the attributes using a sensor device, transmitting the data to a server, and using the data for learning a machine learning model. In Patent Literature 1, when re-learning a machine learning model, the collected data on the missing attributes is added.

[0004] International Publication No. 2022 / 009652

[0005] However, in Patent Literature 1, it is implicitly assumed that the attributes required for the training dataset of the machine learning model do not change, and it is not assumed that the conditions related to the missing attributes will change during the operation of the machine learning model. Therefore, in a situation where the attributes to be collected in the training dataset change, such as in a manufacturing process where the manufacturing method is frequently changed, it is difficult to improve the performance of the machine learning model.

[0006] In view of the above circumstances, the present invention proposes a technique for dynamically identifying and collecting attribute information that should be collected in a training dataset for a machine learning model.

[0007] In order to solve the above problem, the present invention proposes, as an example, a data processing system comprising at least one evaluation device and a computer system that applies first information output from the evaluation device to a learning model and outputs second information, wherein the computer system performs the following processes: a process of calculating third information, which is an evaluation value, from at least one of the first information or the second information; a process of performing statistical operations on multiple pieces of third information to calculate statistics and determining a collection range for collecting the third information based on the statistics; a process of collecting data included in the collection range of the third information from the multiple pieces of third information calculated from the output of the at least one evaluation device; and a process of relearning the learning model using the collected third information and the first information and second information corresponding to the collected third information.

[0008] According to the present invention, it is possible to more reliably manage machine learning models. Specifically, it is possible to maintain the performance of a machine learning model even when attributes to be collected in a training dataset change due to a change in manufacturing method, etc. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0009] FIG. 1 is a diagram showing an outline of changes in measurement value distribution assumed by the present invention. FIG. 2 is a diagram showing an example of the configuration of the present invention. FIG. 3 is a diagram showing the configuration of a storage unit. FIG. 4 is a diagram showing an example of the configuration of a collection range determination unit. FIG. 5 is a diagram showing a flowchart of the collection range determination unit. FIG. 6 is a diagram showing an outline of a statistics calculation unit. FIG. 7 is a diagram showing an example of the configuration of a collection method instruction unit. FIG. 8 is a diagram showing a flowchart of the collection method instruction unit. FIG. 9 is a diagram showing an outline of relevance evaluation using wafer coordinates. FIG. 10 is a diagram showing an outline of relevance evaluation using an apparatus ID. FIG. 11 is a diagram showing an example of a screen of the present invention.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings, although detailed descriptions of overlapping portions may be omitted.

[0011] An embodiment of the present invention assumes a manufacturing line for manufacturing products such as semiconductors. In a semiconductor manufacturing line, products are processed using manufacturing equipment, and the product state is evaluated using an evaluation device. For example, in the case of a semiconductor manufacturing line, manufacturing equipment is equipment such as an etcher or exposure tool, and there are multiple devices performing the same process. Furthermore, evaluation equipment is an inspection and measurement device using optical inspection or a scanning electron microscope (SEM), and there are also multiple devices performing the same inspection or measurement process. Furthermore, products may be finished products or products in the middle of the manufacturing line. In recent years, the application of machine learning models has been increasing in the inspection and measurement processes performed by evaluation devices. These include, for example, models for inspecting products for foreign particles or defects, models for measuring products to evaluate whether they are processed according to design information, and models for performing image processing of evaluation images, such as image quality improvement models that convert low-quality images into high-quality images to facilitate inspection and measurement.

[0012] During the prototype stage of a product, the manufacturing conditions used by the manufacturing equipment are frequently changed to improve yield. The materials used in the product may also be changed. Such changes in manufacturing conditions may also require changes to the evaluations performed by the evaluation equipment. For example, changes in manufacturing conditions or product materials may change manufacturing accuracy, resulting in changes to the distribution of measurement values, etc.

[0013] <Necessity for Re-learning: Using Measurement Values ​​as an Example> Figure 1 is a diagram showing an overview of changes in measurement value distributions. The horizontal axis represents measurement values ​​obtained by measurement. The vertical axis represents the probability density of measurement values ​​obtained by multiple measurements. Measurement value distribution 101 is the measurement value distribution under initial manufacturing conditions. Here, the distribution is near the target design value. Measurement value distribution 102 is the measurement value distribution after a change in manufacturing conditions or materials. Changing the manufacturing conditions or materials can cause manufacturing accuracy to become unstable, widening the measurement value range. In such cases, a machine learning model trained using measurement value distribution 101 cannot correctly process data outside measurement value distribution 101 but inside measurement value distribution 102. Furthermore, even if measurement values ​​occur in both measurement value distribution 101 and measurement value distribution 102, it is difficult to share training data if the quality of the evaluation image obtained changes significantly due to changes in manufacturing conditions or materials. In such cases, it is necessary to recollect the entire training dataset. The measurement value distribution 103 is a measurement value distribution obtained under manufacturing conditions that enable stable manufacturing. Once the manufacturing conditions and material specifications are finalized and mass production begins, measurement values ​​near the design values ​​become dominant. In such cases, collecting a training dataset that encompasses the ranges of the measurement value distribution 101 and the measurement value distribution 102 is ineffective and costly. Patent Document 1 makes it difficult to address cases where the measurement value range to be collected by the training dataset fluctuates or where sharing training data is difficult due to changes in manufacturing conditions or materials. This also applies to discrete value distributions such as defect distributions, as well as continuous value distributions such as measurement value distributions. For example, when the number of defect types that are likely to occur increases or decreases due to changes in manufacturing methods or materials, the collection conditions for the datasets used for training and verification must be changed. Furthermore, in the case of a machine learning model that performs image processing, similar management is required for the continuous and discrete values ​​related to the image processing to maintain the performance of the machine learning model.

[0014] 2 is a diagram showing an example of the schematic configuration of a data processing system (data collection condition specifying device + evaluation device) 200 according to the present invention. In the present invention, when a relearning determination unit 201 determines that relearning is necessary, a data collection condition specifying unit 204 specifies the conditions for data collection using data from an evaluation device 203 stored in a storage unit (storage device) 202.

[0015] The re-learning determination unit 201 determines whether re-learning is necessary by using a monitoring method for the machine learning model (stored in the storage unit 202), a rule-based method related to manufacturing conditions, etc., or a determination by an operator.

[0016] For example, when a machine learning model monitoring method is used, the distribution of measurement values ​​is monitored as shown in Fig. 1. Here, the measurement value distribution used or set when learning the machine learning model is stored, and is periodically compared with the measurement value distribution, and if there is a deviation of a predetermined amount or more, a decision is made to perform re-learning. Monitoring may also be performed using index values ​​related to the distribution of defect types, the number of occurrences, and images other than the measurement values.

[0017] In a rule-based method for manufacturing conditions, for example, a rule is established that determines that re-learning is necessary when manufacturing conditions or materials change, and if the change in manufacturing conditions is minor, a rule may be established so that re-learning is not required.

[0018] In the judgment by the operator, it is determined that relearning is necessary when the operator determines that relearning should be performed.

[0019] Furthermore, even if re-learning is not required, it may be used to identify data collection conditions when it is determined that learning is necessary.

[0020] Furthermore, the relearning determination unit 201 transmits information about the machine learning model determined to require relearning to the data collection condition specification unit 204. This information includes, for example, the identification information (ID) of the machine learning model, the target manufacturing process, manufacturing conditions, evaluation conditions, etc.

[0021] The data collection condition specification unit 204 is composed of a collection range determination unit 205, a collection method instruction unit 206, and a collection completion determination unit 207. The collection range determination unit 205 determines, as the collection range, values ​​that should be covered by a learning dataset used for training a machine learning model. The collection method instruction unit 206 determines a method for collecting the collection range determined by the collection range determination unit 205 and instructs the evaluation device. The collection completion determination unit 207 determines whether collection of data necessary for training in the memory unit 202 has been completed.

[0022] The data collection condition specifying device including the relearning determination unit 201, the storage unit 202, and the data collection condition specifying unit 204 can be configured by a computer system.

[0023] 3 is a diagram showing an example of information managed by the storage unit 202. The storage unit 202 stores and manages information such as a data ID, the date and time related to evaluation and manufacturing, the evaluation image, the manufacturing device used, the manufacturing conditions used, the evaluation device used, the evaluation conditions used, the measurement values ​​obtained by evaluation, the type of product to be manufactured, the target manufacturing process, the presence or absence of defects and the defect types obtained by evaluation, etc. The data collection condition specification unit 204 uses the information stored in the storage unit 202 to propose conditions and collection methods for data required for learning a machine learning model.

[0024] <Example of Internal Configuration of Collection Range Determination Unit 205> Fig. 4 is a diagram showing an example of the internal configuration of the collection range determination unit 205. The collection range determination unit 205 is composed of a data reading unit 401, an evaluation target extraction unit 402, a statistics calculation unit 403, and a collection range calculation unit 404. The operation of each processing unit will be made clear by the content of the processing of each step shown in the flowchart below.

[0025] 5 is a flowchart for explaining the collection range determination process performed by the collection range determination unit 205. The collection range determination unit 205 starts the flowchart when the relearning determination unit 201 determines that relearning is necessary.

[0026] (i) Step S501 The data reading unit 401 reads data from the storage unit 202. At this time, the data to be read may be limited by using information about the learning target transmitted from the relearning determination unit 201. Here, the data refers to, for example, measurement value data acquired up to now when the evaluation value is a measurement value, and information including the location, type, and number of defects when the evaluation value is a defect.

[0027] (ii) Step S502: The evaluation target extraction unit 402 extracts evaluation target data (which data will be used for learning) to be used in evaluating the collection range from the data read in S501. This is performed based on information about changes in data trends, information about the manufacturing process, manufacturing conditions, and evaluation conditions of the learning target, and manually set information. For example, in the case of information about changes in data trends, data from the date and time when the data trend is estimated to have changed is extracted. For example, a measurement value distribution is calculated for each predetermined period, the deviation from the expected measurement value distribution is evaluated, and the date and time showing a deviation of a predetermined amount or more is estimated as the date and time when the data trend changed. Here, in addition to the measurement value distribution, a defect type distribution, a defect count distribution, or an image-related evaluation value distribution may also be used. Furthermore, in the case of information about the learning target, for example, information from the date and time when the manufacturing conditions, materials, or evaluation conditions in the target manufacturing process were changed may be extracted. Furthermore, the extraction target may be limited by the manufacturing process, manufacturing conditions, materials, and evaluation conditions. In addition to this, manually set information may be used to limit the data to be extracted.

[0028] (iii) Step S503: The statistics calculation unit 403 calculates statistics of the data extracted in step S502. When calculating the statistics, the data distribution is considered. FIG. 6 is a diagram showing an example of a measurement value distribution when the evaluation value is a measurement value. As shown in FIG. 6, for example, values ​​resulting from measurement failure may be mixed into the measurement value distribution. In such cases, it is not realistic to collect data that includes all of the observed measurement values. Therefore, by determining the collection range using statistics such as the q quantile, it is possible to determine a collection range that is robust to noise.

[0029] (iv) Step S504 The collection range calculation unit 404 determines the collection range based on the statistics calculated by the statistics calculation unit 403. This can be done, for example, by using the q quantile to identify a range of measurement values ​​in the distribution, for example, from the top 2% to 98% (not limited to this range (it may be 25% to 75%, for example). The operator can also specify the range), and setting the range to include this range. Using the q quantile makes it possible to set a range that is robust against noise. The range of measurement values ​​may also be set individually for each target.

[0030] When dealing with a discrete value distribution such as a defect type, the collection range may be a certain number of observed discrete values. Furthermore, instead of the range of continuous or discrete values ​​to be collected, a value corresponding to the number of data to be collected may be calculated. This may be a preset value, or the collection range may be divided into several ranges like a histogram (e.g., by further subdividing the above-mentioned 2% to 98% range) and the amount of data to be collected for each range may be calculated. This may be determined based on a range that is expected to allow smooth data collection. For example, it may be determined based on the amount of data that is expected to be collectable when manufacturing and evaluation are performed a predetermined number of times. The collection range may also be calculated using the trend of changes in statistics. For example, if the measurement value distribution gradually becomes wider, a wider measurement value distribution may be acquired in advance for the future. In this case, the collection range may be set by approximating the time change of the measurement values ​​and predicting the future collection range.

[0031] 7 is a diagram showing an example of the internal configuration of the collection method instruction unit 206. The collection method instruction unit 206 can refer to a controllable parameter 701, and is composed of a relevance assessment unit 702, a collection condition determination unit 703, a collection procedure generation unit 704, and a collection instruction unit 705.

[0032] <Contents of Collection Method Instruction Processing> FIG. 8 is a flowchart for explaining the contents of collection method instruction processing by the collection method instruction unit 206.

[0033] (i) Step S801 The relevance evaluation unit 702 uses the data extracted by the evaluation target extraction unit 402 of the collection range determination unit 205 to evaluate the relevance with controllable parameters (parameters that can be specified externally; they can also be called "specifiable parameters") 701. Examples of controllable parameters (specifiable parameters) include coordinate positions on a wafer and equipment IDs. Note that the controllable parameters may be specified by the operator on a GUI (see FIG. 11) described below.

[0034] (ii) Step S802 The collection condition determination unit 703 determines collection conditions that will achieve the target distribution using the evaluation result of the relevance obtained in step S801.

[0035] 9 and 10, an overview of the evaluation of relevance and collection conditions utilizing the relevance will be described. Here, the relevance is, for example, information indicating what evaluation value (measurement value, number of defects, etc.) is likely to be obtained at which location on the wafer (characteristics of the evaluation value at each location). Information such as the fact that measurement values ​​tend to vary at the peripheral portion (edge) of the wafer but are stable at the central portion (middle) or that measurement values ​​tend to overshoot in device A but undershoot in device D and are relatively stable in device B, etc.

[0036] FIG. 9 illustrates an overview of the evaluation of correlation with coordinates on a wafer. Measurement values ​​and other data may vary depending on the coordinate on the wafer. Reference numeral 901 indicates statistical quantities, such as the average measurement value for each wafer coordinate and the average error from the design value, using shades of color. As such, the distribution of measurement values ​​may vary for each wafer coordinate. In such a case, for example, if a target distribution (specifiable by the operator) is determined so that the measurement value distribution is uniform within the collection range determined by the collection range determination unit, the collection range is divided by a predetermined number, and wafer coordinates to which the statistical quantities calculated in step S801 belong are determined for each divided collection range. Then, wafer coordinates to be collected are determined so that the number of collected data is equal for each divided collection range. This allows the collection conditions to be determined so that the distribution is uniform within the collection range. Reference numeral 902 indicates the collection conditions determined in this manner. Note that while an example of a uniform target distribution is described here, other examples, such as a normal distribution, can also be used to determine the collection conditions. For example, in the case of a normal distribution, the required number of data points for each measurement value range is determined, and then wafer coordinates highly correlated with each measurement value range are acquired to achieve that number of data points, and these are used as collection conditions. This method makes it possible to determine collection conditions that achieve any desired distribution. Note that statistics refer to values ​​used to evaluate relevance. For example, the aforementioned range of 2% to 98% corresponds to a statistic (the same concept as the q-quantile, etc.). Furthermore, the above-mentioned "determining wafer coordinates to be collected so that the number of data points is equal for each divided collection range" should be considered in order to use a uniform distribution in machine learning. For example, if there are 100 measurement values ​​between m and 2 m nanometers and only 10 between 2 m and 3 m nanometers, the learning results will be more influenced by the measurement values ​​between m and 2 m nanometers than by the measurement values ​​between 2 m and 3 m nanometers. Therefore, it is desirable for the data used for learning to have a uniform distribution (the same number of each measurement value) (it is desirable for each measurement value (each evaluation value) to be the same number). Therefore, it is important to divide the collection range to achieve a uniform distribution. For example, in Figure 6, outliers at the ends are excluded, but if there are still a large number of measurement values, additional data can be collected to ensure the number of data points is the same, or some data can be omitted from learning.

[0037] FIG. 10 is a diagram showing the evaluation of the relevance with the equipment ID. In FIG. 10, the vertical axis represents the value used for the evaluation of the relevance. For example, when the evaluation value is a length measurement value, the length measurement value observed for each equipment may differ. In the case of FIG. 10, if a high evaluation value is to be collected, the evaluation value (length measurement value) is collected from equipment A, and if a low evaluation value is to be collected, the evaluation value (length measurement value) is collected from equipment B. Here, the equipment ID may be an equipment ID related to the manufacturing equipment or an equipment ID related to the evaluation equipment.

[0038] As described above, FIGS. 9 and 10 illustrate the relevance evaluation when wafer coordinates and equipment IDs are used as controllable parameters 701. By evaluating the relevance with the controllable parameters 701 in this manner, it is possible to identify the conditions under which data should be collected to achieve a target distribution of evaluation values. The controllable parameters 701 may be values ​​other than wafer coordinates and equipment IDs. In the present invention, controllable parameters can be used when determining collection conditions. Values ​​other than measurement values ​​may also be used to evaluate the relevance. For example, defect types or defect counts may be used. In this case, it is possible to identify wafer coordinates or equipment where specific defect types are likely to occur and to perform focused data collection. Image-related evaluation values ​​may also be used. Furthermore, although FIGS. 9 and 10 illustrate an example in which collection conditions are determined using a single controllable parameter, relevance may also be evaluated using a combination of multiple controllable parameters. For example, relevance evaluation may be performed for each combination of equipment and wafer coordinates. Furthermore, although it has been described that collection conditions are determined using highly relevant controllable parameters, collection conditions may also be set regardless of relevance. For example, collection conditions may be set to include a condition for collecting data from the entire surface of the wafer coordinates regardless of relevance, or multiple equipment IDs.

[0039] To summarize the above, the collection conditions are determined by the following procedure: First, requirements regarding the collection range are determined. This is the target number of data to be collected within the collection range or within each divided collection range.

[0040] Next, for each collection range or divided collection range, a controllable parameter with a high evaluation value of relevance is identified. For example, if the collection range is related to the length measurement value, this is performed by identifying a controllable parameter that has the possibility of observing the divided collection range, i.e., the divided range of the length measurement value.

[0041] Next, the controllable parameters with high relevance identified for each collection range or divided collection range are selected and added to the collection conditions. For example, if the collection range is related to measurement values, this is an operation of adding controllable parameters that may observe the divided collection ranges, i.e., the divided ranges of measurement values, to the collection conditions. One or more controllable parameters may be specified for each collection range or divided collection ranges.

[0042] Finally, collection conditions may be added according to pre-set conditions, such as adding a predetermined number of wafer coordinates to the collection conditions or adding multiple device IDs to the collection conditions.

[0043] As a result, the collection conditions include controllable parameters to be collected by the evaluation device in order to collect the required number of data points for each collection range or each divided collection range.

[0044] (iii) Step S803: The collection procedure generation unit 704 generates a collection procedure according to the collection conditions determined in step S802. Here, the collection procedure refers to a procedure for evaluating a product using the evaluation device 203 and collecting data (which may also be information indicating where the data is to be collected). This collection procedure is generated to include the controllable parameters 701 set as the collection conditions in step S802. That is, the collection procedure generation unit 704 may create a collection procedure for each product or process to be collected, including the controllable parameters 701, such as wafer coordinates and equipment IDs, set by the relevance evaluation, and then create a collection procedure such that the parameters, such as wafer coordinates and equipment IDs, satisfy predetermined conditions regardless of the relevance evaluation. Here, the predetermined conditions may include, for example, including wafer coordinates at predetermined locations or including a predetermined number or more of manufacturing devices or evaluation devices.

[0045] Furthermore, the collection procedure may be created by using the collection procedure used in the initial data collection and editing the items for which new collection conditions are set. That is, in the existing collection procedure, it is conceivable to edit only the wafer coordinates to be imaged, or to change the distribution device, etc.

[0046] (iv) Step S804: The collection instruction unit 705 distributes the information on the collection procedure generated in step S803 to the evaluation device 203 (the distribution destination can be specified by the operator). At this time, if the collection conditions are set in step S802 to include multiple manufacturing devices or evaluation devices, the collection instruction unit 705 instructs the evaluation device 203 to collect data from the set manufacturing devices or evaluation devices. That is, when collecting data on products processed by multiple manufacturing devices, the collection instruction unit 705 may distribute the IDs of the manufacturing devices and collection targets of the collection conditions to the multiple evaluation devices in association with the collection procedure, and collect data when a product such as a wafer associated with the collection target and the ID of the manufacturing device is input into the evaluation device. If the collection conditions are set to include data from multiple evaluation devices, the collection procedure and collection targets may be distributed to the evaluation device, and data collection may be performed when data from the collection targets is observed.

[0047] The collection completion determination unit 207 also determines whether data collection has been completed in accordance with a predetermined criterion within the collection range determined by the collection range determination unit 205. This is, for example, a process of dividing the collection range into several subranges and determining whether a predetermined amount of data has been collected for each subrange. Here, the predetermined amount for each subrange refers to an amount determined so that the entire collection range has a target distribution shape, such as a uniform distribution or a normal distribution. If a predetermined amount (predetermined number) or more of data (evaluation values) can be collected for each subrange, the learning model may be trained (retrained) using output data from the evaluation device 203 corresponding to the collectable data and data obtained by applying the output data to the learning model.

[0048] In addition, even if not included in the collection conditions, the wafer coordinates, equipment ID, etc. may be evaluated to see if they meet predetermined criteria, and the process may be determined to be complete only if they do. This allows the learning data set to be constructed so that it meets the specified conditions. For example, it is possible that not enough data is collected in the edge area of ​​the wafer. In such a case, if data has been collected from the target equipment and the operator determines that it is acceptable, the data may be used as is even if there is a slight shortage of learning data.

[0049] Furthermore, the collection completion determination by the collection completion determination unit 207 and the determination of the collection range by the collection range determination unit 205 may be performed alternately. This allows the collection range to be corrected (e.g., expanded) using additional data collected while collecting data within the initially determined collection range. Furthermore, if there are conditions that make collection difficult, the operator may be presented with the difficult conditions. The operator can consider creating data for the displayed difficult conditions and adding it to the learning data. For example, if there is a shortage of data for a specific measurement value, the manufacturing conditions may be changed so that data for that measurement value can be collected. Furthermore, a display unit may be provided to display the data collection status to the operator (see the collection condition display unit 1102 and the collection procedure display unit 1_1103 in FIG. 11). This allows the operator to check the data collection status and determine whether or not to perform learning.

[0050] 11 is a diagram showing an example of the configuration of an operator confirmation screen (GUI: Graphical User Interface) 1100. The operator confirmation screen 1100 includes a target display section 1101, a collection condition display section 1102, a collection procedure display section 1103, a collection procedure display section 2104, a collection procedure confirmation section 1105, and a collection execution instruction section 1106.

[0051] The target display section 1101 is an area for displaying information about the machine learning model to be trained and information about the target to be evaluated for the training data.

[0052] The collection condition display section 1102 is an area that displays the collection range determined by the collection range determination section 205 and the collection conditions determined by the collection condition determination section 703. This allows the operator to check the required collection range and the degree to which the collection range is satisfied.

[0053] The collection procedure display section 1_1103 and the collection procedure display section 2_1104 are areas that display information for confirming the proposed collection procedure. The collection procedure display section 1_1103 displays the collection procedure related to wafer coordinates. The wafer coordinates for which collection is proposed may be displayed together with the statistical quantities for each wafer coordinate that serve as the basis. The collection procedure display section 2_1104 displays the collection procedure related to the equipment ID, highlighting the equipment ID for which collection is proposed. Here too, the statistical quantities for each equipment ID that serve as the basis may be displayed together.

[0054] The collection procedure confirmation section 1105 is an area that displays a button to be pressed when confirming the collection procedure. When the operator presses the button when he / she wants to confirm the collection procedure, the created collection procedure is displayed and the operator is prompted to confirm it.

[0055] The collection execution instruction section 1106 is an area that displays a button to be pressed when executing data collection. When the operator presses the button to instruct execution of data collection, information on the collection procedure is distributed to each evaluation device, and data collection is executed in the device to which the information is distributed.

[0056] <Modifications> The above embodiment describes data collection for training a specific model. However, the present invention may also be applied to cases where multiple models are trained simultaneously. That is, cases where different models are trained for each manufacturing process or evaluation condition. In such cases, the behavior of the data collection condition specification unit 204 may be changed using information about the learning target of the machine learning model. Specifically, possible modifications include changing the conditions for data extracted by the evaluation target extraction unit 402, controlling the application target of the collection procedure generated by the collection procedure generation unit 704 using information about the learning target, and executing the collection completion determination unit 207 for each machine learning model.

[0057] Furthermore, when re-learning, the operator can identify the missing data and collect only the missing data, or can discard the existing data and collect new data. If the image quality of the evaluation images changes significantly due to changes in product materials, for example, past data may hinder learning. In such cases, simply collecting the missing data will not allow for training of a high-performance machine learning model. For this reason, all data may be collected again. Furthermore, to automate these decisions, a mechanism may be introduced that evaluates fluctuations in the image quality of the evaluation images and re-collects all data if there is a certain level of fluctuation.

[0058] Furthermore, information necessary for determining the target machine learning model may be stored in the storage unit 202 and used to control the behavior of each machine learning model during collection.

[0059] <Effects of the Present Embodiment> According to the present embodiment, it is possible to efficiently collect data necessary for training a machine learning model and build a highly reliable model. Furthermore, according to the present embodiment, when training of a machine learning model is required, it is possible to create collection conditions and collection procedures for collecting a training dataset.

[0060] Here, the collection conditions refer to the data values ​​that a training dataset must encompass in order to train a highly reliable machine learning model. A training dataset that collects a predetermined amount of data for each value is expected to include the attributes of data that can be input to the machine learning model, thereby preventing the machine learning model from producing unexpected output. Furthermore, the collection procedure refers to a procedure for collecting data aimed at achieving the collection conditions (e.g., a recipe indicating from which locations data should be collected). This corresponds to data describing parameters for controlling the behavior of an evaluation device. For example, the procedure describes the procedure for acquiring evaluation images for each collection target that is the subject of training for the machine learning model. The procedure includes information about the product being evaluated, such as imaging conditions and wafer coordinates.

[0061] These collection conditions and collection procedures may be presented to the operator to prompt confirmation, or if predetermined conditions are met, distribution and collection may be performed automatically to the device.

[0062] Furthermore, the configuration of the present invention enables an operator to efficiently manage the quality of training data for a machine learning model. By checking the example screen of the present invention, the operator can confirm the attributes of the data to be collected and their value ranges, check the current collection status, and check the proposed collection procedure, thereby confirming the collection status of the training dataset. Furthermore, if there is data that is difficult to collect among the missing data, the operator may collect the training data by intentionally performing operations to generate the missing data. This corresponds to, for example, an operation such as changing design information when data for a specific measurement value is missing, so that the missing measurement value data can be manufactured.

[0063] <Others> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to facilitate understanding of the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace a part of a certain configuration example with another configuration example, and it is also possible to add another configuration example to one configuration example. Furthermore, it is possible to add, delete, or replace other configurations with respect to part of the configuration example.

[0064] The functions of the above-described embodiments can also be realized by software program code. In this case, a storage medium on which the program code is recorded is provided to a system or device, and the computer (or CPU or MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.

[0065] Furthermore, an operating system (OS) running on a computer may perform some or all of the actual processing based on the instructions of the program code, and the functions of the above-described embodiments may be realized by this processing.Furthermore, after the program code is read from a storage medium and written to a memory on the computer, a CPU of the computer may perform some or all of the actual processing based on the instructions of the program code, and the functions of the above-described embodiments may be realized by this processing.

[0066] In addition, the program code of the software that realizes the functions of this embodiment may be distributed via a network and stored in a storage means such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R, so that when used, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage means or storage medium.

[0067] The processes and techniques described herein are not inherently related to any specific device and can be implemented by a combination of components. Various types of general-purpose devices can also be added. A dedicated device may be constructed to perform the functions of this embodiment. Various functions can also be formed by appropriately combining multiple components disclosed in this embodiment. For example, some components may be omitted from all the components shown in the embodiments and examples, or components from different examples may be appropriately combined.

[0068] Although specific embodiments are described herein, they are in all respects illustrative and not limiting. Those skilled in the art will recognize that there are numerous combinations of hardware, software, and firmware suitable for implementing the present invention. For example, the described software can be implemented in a wide variety of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, Java, etc.

[0069] Furthermore, in the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0070] Additionally, other implementations of the present invention will be apparent to those skilled in the art from consideration of the present embodiments. The specification and examples are exemplary only, with the scope and spirit of the invention being indicated by the following claims.

[0071] 101 Measurement value distribution under initial conditions 102 Measurement value distribution after change of manufacturing conditions 103 Measurement value distribution during mass production 200 Data processing system 201 Re-learning determination unit 202 Storage unit 203 Evaluation device 204 Data collection condition specification unit 205 Collection range determination unit 206 Collection method instruction unit 207 Collection completion determination unit 401 Data reading unit 402 Evaluation target extraction unit 403 Statistical quantity calculation unit 404 Collection range calculation unit 701 Controllable parameter 702 Relevance evaluation unit 703 Collection condition determination unit 704 Collection procedure generation unit 705 Collection instruction unit 1100 Operator confirmation screen 1101 Target display unit 1102 Collection condition display unit 1103 Collection procedure display unit 1 1104 Collection procedure display unit 2 1105 Collection procedure confirmation unit 1106 Collection execution instruction section

Claims

1. A data processing system comprising at least one evaluation device and a computer system that applies first information output from the evaluation device to a learning model and outputs second information, wherein the computer system performs the following processes: a process of calculating third information, which is an evaluation value, from at least one of the first information or the second information; a process of performing statistical operations on multiple pieces of third information to calculate statistics and determining a collection range for collecting the third information based on the statistics; a process of collecting data included in the collection range of the third information from the multiple pieces of third information calculated from the output of the at least one evaluation device; and a process of relearning the learning model using the collected third information and the first information and second information corresponding to the collected third information.

2. A data processing system as claimed in claim 1, wherein the computer system divides the collection range into a plurality of sub-ranges, and when a predetermined number or more of the third information can be collected for each sub-range, re-learns the learning model using the first information and the second information corresponding to the third information that can be collected.

3. A data processing system according to claim 1, wherein the computer system performs a process of evaluating the relevance between the third information and a parameter that can be specified and is used in the evaluation performed by the at least one evaluation device, and a process of determining collection conditions for data included in the collection range using the results of the relevance evaluation so as to achieve a specified target distribution.

4. A data processing system as set forth in claim 2, wherein the computer system executes a process of evaluating the relevance between the third information and a parameter that can be specified and is used in the evaluation performed by the at least one evaluation device, and a process of determining, using the results of the relevance evaluation, the collection conditions for data included in the collection range or the sub-range so as to achieve a specified target distribution.

5. A data processing system according to claim 4, wherein a plurality of types of parameters are designated, and the computer system determines the collection conditions so that a predetermined number or more of data for each of the plurality of types of parameters is included in the collection range or sub-range.

6. A data processing system according to claim 4, wherein the computer system executes the following processes: generating a data collection procedure in accordance with the collection conditions so as to include the parameters set as the collection conditions; distributing the data collection procedure to the at least one evaluation device; and collecting the first information acquired according to the distributed collection procedure, and the second information and third information related to the first information acquired according to the distributed collection procedure.

7. A data processing system according to claim 6, wherein the at least one evaluation device collects the first information acquired in accordance with the distributed collection procedure, and the second information and the third information related to the first information acquired in accordance with the distributed collection procedure, for products processed by at least one manufacturing device included in the parameters.

8. A data processing system according to claim 7, wherein the computer system executes a process of displaying on a display screen whether or not a predetermined number of data items have been collected within the collection range or the small range.

9. A data processing method for training a learning model that outputs second information by applying first information output from at least one evaluation device using a computer, the data processing method comprising: calculating third information, which is an evaluation value, from at least one of the first information or the second information; performing statistical operations on a plurality of pieces of third information to calculate statistics and determining a collection range for collecting the third information based on the statistics; collecting data included in the collection range of the third information from the plurality of pieces of third information calculated from the output of the at least one evaluation device; and re-training the learning model using the collected third information and the first information and second information corresponding to the collected third information.

10. A data processing method as set forth in claim 9, wherein the computer divides the collection range into a plurality of sub-ranges, and if a predetermined number or more of the third information can be collected for each sub-range, re-learns the learning model using the first information and the second information corresponding to the third information that can be collected.

11. A data processing method according to claim 9, further comprising: evaluating the relevance between a parameter that is used in the evaluation performed by said at least one evaluation device and that can be designated, and said third information; and using the results of said relevance evaluation, determining collection conditions for data included in said collection range so as to achieve a designated target distribution.

12. A data processing method according to claim 10, further comprising: evaluating the relevance between the third information and a parameter that can be specified and is used in the evaluation performed by the at least one evaluation device; and using the result of the relevance evaluation, determining collection conditions for data included in the collection range or the sub-range so as to achieve a specified target distribution.

Images