Data processing systems and data processing methods

Abstract

In order to dynamically specify and collect the attribute information to be collected in the learning data set of a machine learning model, the present invention proposes a data processing system, which performs the following steps: calculating an evaluation value, i.e., a third information, from at least one of the first information or the second information; calculating a statistical quantity by performing statistical calculations on a plurality of third information items, and determining the collection range for collecting the aforementioned third information items based on the statistical quantity; collecting data that will be included in the collection range of the third information from the plurality of third information items calculated based on the output of at least one evaluation device; and relearning the learning model using the collected third information and the first and second information corresponding to the collected third information.

Description

[Technical Field]

[0001] This invention relates to a data processing system and a data processing method. [Previous Technology]

[0002] The performance of a machine learning model depends on the quality of the training data. The quality of the training data depends on the comprehensiveness of the attributes of the data that may be input into the machine learning model. Therefore, it is desirable to collect the training data in a way that encompasses the diverse variables that may be input into the model.

[0003] For example, Patent Document 1 discloses that attributes that are deficient during learning are specified, data on these attributes are collected in a sensor device, and sent to a server for use in the learning of a machine learning model. In Patent Document 1, during the relearning of the machine learning model, the collected data on deficient attributes are added. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] International Publication No. 2022 / 009652 [Summary of the Invention]

[0005] [Problem to be Solved by the Invention] However, Patent Document 1 implicitly assumes that the required attributes of the learning dataset of the machine learning model will not change, and does not consider the fact that the conditions for insufficient attributes will gradually change during the use of the machine learning model. Therefore, it is difficult to improve the performance of the machine learning model when the attributes to be collected in the learning dataset change due to frequent changes in the manufacturing method, etc.

[0006] In view of the above situation, this disclosure proposes a technique for dynamically specifying and collecting the attribute information that should be collected from the learning dataset of a machine learning model. [Means for solving the problem]

[0007] To address the aforementioned issues, as an example, this invention proposes a data processing system comprising at least one evaluation device and a computer system that applies first information output from the evaluation device to a learning model to output second information. The aforementioned computer system performs the following processes: calculating an evaluation value, i.e., third information, from at least one of the aforementioned first information or the aforementioned second information; calculating a statistical measure based on a plurality of the aforementioned third information, and determining a collection scope for collecting the aforementioned third information based on the statistical measure; collecting data from the plurality of the aforementioned third information calculated based on the output of the aforementioned at least one evaluation device, that is, data included within the aforementioned collection scope of the aforementioned third information; and relearning the aforementioned learning model using the previously collected previously third information, and the previously collected previously first and previously second information. [Effects of the Invention]

[0008] According to the present invention, machine learning models can be managed with greater reliability. Specifically, even if the attributes to be collected in the learning dataset change due to changes in the manufacturing method, the performance of the machine learning model can still be maintained. The issues, structures, and effects mentioned above will become clear from the following description of the embodiments.

Implementation Method

[0010] The following uses drawings to illustrate embodiments of the present invention. In addition, detailed descriptions of repeated parts are sometimes omitted.

[0011] An embodiment of the present invention is envisioned as a production line for manufacturing products such as semiconductors. In a semiconductor production line, products are processed by manufacturing equipment, and the product condition is evaluated by an evaluation device. For example, in the case of a semiconductor production line, the manufacturing equipment is an etching machine or an exposure device, and multiple such devices are present. Similarly, the evaluation device is an inspection and measurement device utilizing optical inspection or a scanning electron microscope (SEM), and multiple such devices are present. Furthermore, the product can be a finished product or a product in the middle of the production line. In recent years, the application of machine learning models in the inspection / measurement processes performed in evaluation devices has been gradually increasing. These include, for example, models for inspecting foreign objects or defects mixed into products, models for measuring whether products have been processed according to design information, and image processing models for evaluating images, such as models for converting low-quality images into high-quality images to facilitate inspection and measurement.

[0012] During the product trial production phase, the manufacturing conditions used in the manufacturing equipment are frequently changed in order to improve yield. Furthermore, the materials used in the product are sometimes changed. There are also situations where changes in manufacturing conditions necessitate changes in the evaluation performed in the evaluation equipment. For example, sometimes changes in manufacturing conditions or product materials can lead to changes in manufacturing precision or variations in the distribution of measurements such as length.

[0013] <On the Necessity of Relearning: Taking Length Measurement Values ​​as an Example> Figure 1 is a diagram illustrating the general changes in the length measurement value distribution. The horizontal axis represents the length measurement values ​​obtained through measurement. The vertical axis represents the probability density of length measurement values ​​obtained through multiple measurements. Length measurement value distribution 101 is the length measurement value distribution caused by the initial manufacturing conditions. Here, it is distributed near the design value that serves as the target. Length measurement value distribution 102 is the length measurement value distribution after changes in manufacturing conditions or materials. When manufacturing conditions or materials are changed, manufacturing accuracy becomes unstable, and the range of length measurement values ​​widens. In such cases, a machine learning model that has learned using length measurement value distribution 101 cannot correctly process data that falls outside length measurement value distribution 101 but inside length measurement value distribution 102. Furthermore, even for length values ​​occurring in both length value distributions 101 and 102, if the image quality of the evaluation image changes significantly due to changes in manufacturing conditions or materials, it becomes difficult to share learning data. In such cases, the entire learning dataset must be recollected. Length value distribution 103 is a length value distribution obtained under stable manufacturing conditions. When the specifications of manufacturing conditions or materials are determined and mass production begins, length values ​​near the design value will dominate. In such cases, if the learning dataset is collected in a way that encompasses the range of length value distributions 101 or 102, the effect is weak and the cost is high. Patent Document 1 cannot adequately address situations where the range of length values ​​to be collected in the learning dataset changes, or where it becomes difficult to share learning data due to changes in manufacturing conditions or materials. Moreover, this applies not only to continuous value distributions like length value distributions but also to discrete value distributions like defect distributions. For example, if the types of defects that are prone to occur increase or decrease due to changes in manufacturing methods or materials, the data collection conditions used during learning / verification must be changed. Similarly, in the case of machine learning models performing image processing, the continuous and discrete values ​​related to that image processing must also be managed in the same way to maintain the performance of the machine learning model.

[0014] <System Configuration Example> Figure 2 is a schematic diagram of a data processing system (data collection condition specification device + evaluation device) 200 resulting from the present invention. In the present invention, when it is determined by the relearning determination unit 201 that relearning is required, the data stored in the memory unit (memory device) 202 by the evaluation device 203 is used to specify the conditions for the data to be collected by the data collection condition specification unit 204.

[0015] The relearning determination unit 201 determines the necessity of relearning by means of monitoring the machine learning model (which has been stored in the memory unit 202), or by means of rules based on manufacturing conditions, or by determinations made by the operator.

[0016] For example, when using a machine learning model for monitoring, the distribution of measurement values, as shown in Figure 1, is monitored. Here, the distribution of measurement values ​​used or pre-set during the learning process of the machine learning model is memorized, and it is periodically compared with the distribution of measurement values. If there is a deviation of a certain value or more, a decision is made to relearn. In addition to measurement values, the distribution or occurrence number of defect types and image-related index values ​​can also be used for monitoring.

[0017] In the method of establishing rules based on manufacturing conditions, for example, rules that determine the need for relearning when manufacturing conditions or materials change are constructed. At this time, rules can also be constructed so that relearning is not performed if the change in manufacturing conditions is minor.

[0018] In operator-caused determinations, when it is determined by the operator that relearning is required, it is considered that relearning is required.

[0019] Furthermore, even if it is not for further learning, it can be used to collect specific data when it is determined that learning is required.

[0020] Furthermore, the relearning determination unit 201 sends information about machine learning models that have been determined to require relearning to the data collection condition specific unit 204. This information may include, for example, the machine learning model's identification information (ID), or the manufacturing process or manufacturing conditions, evaluation conditions, etc., of the object.

[0021] The data collection condition specification unit 204 comprises a collection scope determination unit 205, a collection method instruction unit 206, and a collection completion determination unit 207. The collection scope determination unit 205 determines the collection scope by considering the values ​​to be collected from the learning dataset used by the machine learning model during learning. The collection method instruction unit 206 determines the method required to collect the data within the scope determined by the collection scope determination unit 205 and instructs the evaluation device accordingly. The collection completion determination unit 207 determines whether the memory unit 202 has completed the collection of the data necessary for learning.

[0022] Furthermore, the data collection condition specification device, which includes a relearning determination unit 201, a memory unit 202, and a data collection condition specification unit 204, can be configured by a computer system.

[0023] <Example of Information Management (Storage)> Figure 3 is an illustration of an example of information managed in the memory unit 202. The memory unit 202 stores and manages information such as: data ID, date and time of evaluation or manufacturing, evaluation image, manufacturing equipment used, manufacturing conditions used, evaluation equipment used, evaluation conditions used, measurement value obtained by evaluation, product type of the manufactured object, manufacturing process, presence or type of defects obtained by evaluation, etc. The data collection condition specification unit 204 uses the information stored in the memory unit 202 to propose the conditions or collection methods of the data required for the learning of the machine learning model.

[0024] <Example of the Internal Structure of the Collection Scope Determination Unit 205> Figure 4 is a diagram illustrating an example of the internal structure of the collection scope determination unit 205. The collection scope determination unit 205 consists of a data input unit 401, an evaluation object extraction unit 402, a statistical calculation unit 403, and a collection scope calculation unit 404. The operations of each processing unit can be understood by referring to the processing steps shown in the flowchart below.

[0025] <Content of Collection Scope Determination Process> Figure 5 is a flowchart illustrating the collection scope determination process initiated by the collection scope determination unit 205. The collection scope determination unit 205 begins the flowchart when the collection scope determination unit 201 determines that relearning is required.

[0026] (i) In step S501, the data reading unit 401 reads data from the memory unit 202. At this time, the reading object can also be limited using the information about the learning object sent from the relearning determination unit 201. Here, the data refers to, for example, the length value data obtained so far when the evaluation value is a length value, and the information on the location, type, and number of defects when the evaluation value is a defect.

[0027] (ii) Step S502 Evaluation Object Extraction Unit 402 extracts the evaluation object data (which data will be used for learning) used during the evaluation of the collection scope from the data read in S501. This is done by: information about changes in data trends, or information about the manufacturing process or manufacturing conditions / evaluation conditions of the learning object, or information set manually. For example, if it is information about changes in data trends, data after the date and time on which the data trend is presumed to have changed will be extracted. For example, the length value distribution can be calculated at fixed intervals, the deviation from the desired length value distribution can be evaluated, and the date and time on which the deviation is above the predetermined date can be presumed as the date and time on which the data trend has changed. Here, in addition to the length value distribution, the defect type distribution or defect number distribution, and the evaluation value distribution of the image can also be used. Furthermore, for example, if the information pertains to a learning object, information from the date and time after which the manufacturing conditions, materials, and evaluation conditions of the object's manufacturing process were changed can be extracted. Additionally, the extracted object can be limited by the manufacturing process, manufacturing conditions, materials, and evaluation conditions. Furthermore, information set manually can also be used to limit the data of the extracted object.

[0028] (iii) Step S503 The statistic calculation unit 403 calculates the statistics of the data extracted in step S502. When calculating the statistics, the data distribution is taken into account. Figure 6 is an illustration of an example of the distribution of length values ​​when the evaluation value is a length value. As shown in Figure 6, for example, in the length value distribution, values ​​such as those from length measurement failures may sometimes be mixed in. In such cases, it is not practical to collect data in a way that includes all length values ​​that have been observed. Therefore, by using statistics such as q quantiles to determine the collection range, a collection range that is more robust to noise can be determined.

[0029] (iv) Step S504: The collection range calculation unit 404 determines the collection range based on the statistics calculated in the statistics calculation unit 403. This is achieved by using, for example, the q-quantile, to specify a range of measurement values ​​in the aforementioned distribution, for example, from the upper 2% to 98% (not limited to this range (it could also be from 25% to 75%, etc.). Alternatively, the operator can specify the range), and the range is set in a manner that includes this range. By using the q-quantile, a range setting that is robust against noise can be achieved. Furthermore, the range of measurement values ​​can also be set individually for each object.

[0030] Furthermore, in the case of discrete value distributions such as operational defects, a certain number or more of the observed discrete values ​​can be used as the collection range. Also, it is possible to calculate not only the range of continuous and discrete values ​​to be collected, but also the value corresponding to the number of data points to be collected. This can be done using pre-set values, or by dividing the collection range into several ranges (for example, further subdividing the aforementioned range from 2% to 98%), and calculating the amount of data to be collected in each range. This can also be decided within a range where data collection is considered to be feasible. For example, it can be determined based on the amount of data considered to be collectable during a predetermined number of manufacturing and evaluation operations. Furthermore, the collection range can also be calculated using the trend of changes in statistical quantities. For example, in cases where the length measurement value distribution gradually becomes wider, a wider range of length measurement value distributions can be obtained in advance to prepare for the future. In this case, the time variation of the measured value can be approximated, and the future collection range can be predicted and then set as the collection range.

[0031] <Example of the internal structure of the collection method instruction unit 206> Figure 7 is a diagram illustrating an example of the internal structure of the collection method instruction unit 206. The collection method instruction unit 206, which can be referenced to the control possible parameter 701, is composed of a correlation evaluation unit 702, a collection condition determination unit 703, a collection procedure generation unit 704, and a collection instruction unit 705.

[0032] <Content of Collection Method Instruction Processing> Figure 8 is a flowchart explaining the content of collection method instruction processing caused by the collection method instruction unit 206.

[0033] (i) Step S801, the correlation evaluation unit 702, uses the data extracted by the evaluation object extraction unit 402 of the collection range determination unit 205 to perform a correlation evaluation with the controllable parameters (parameters that can be specified from the outside; or, in other words, specified possible parameters) 701. Examples of controllable parameters (specified possible parameters) include, for example, the coordinate position on the wafer or the device ID. In addition, the controllable parameters can also be specified by the operator on the GUI (see Figure 11) described later.

[0034] (ii) Step S802 Collection Condition Determination Unit 703 uses the correlation evaluation result obtained in step S801 to determine the collection conditions for the distribution that can achieve the goal.

[0035] Referring to Figures 9 and 10, an overview of the evaluation of correlation and the collection conditions utilizing correlation will be explained. Here, correlation refers to, for example, information indicating which location on the wafer is most likely to yield what kind of evaluation value (length measurement or defect count, etc.) (characteristics of evaluation values ​​at different locations). This includes information such as: length measurement values ​​tend to be inconsistent at the periphery (end) of the wafer, while length measurement values ​​are more stable at the center (the very center); or length measurement values ​​may be higher in device A, lower in device D, and more stable in device B, etc.

[0036] Figure 9 is a diagram illustrating the general overview of the correlation evaluation with coordinates on the wafer. Measurement values, etc., sometimes deviate from the coordinates on the wafer. Symbol 901 represents the statistical quantity such as the average error between the measurement value and the average value or design value for each wafer coordinate using color intensity. Thus, the distribution of measurement values ​​changes with each wafer coordinate. In such cases, for example, when determining the target distribution (which can be specified by the operator) so that the measurement values ​​are uniformly distributed within the collection range determined by the collection range determination unit, the collection range is divided into a pre-determined number, and the wafer coordinates to which the statistical quantity calculated in step S801 belongs are determined for each divided collection range, so that the same number of values ​​are present in each divided collection range. This allows the collection conditions to be determined in a way that results in a uniform distribution within the collection range. Symbol 902 indicates the collection conditions thus determined. Furthermore, while an example of a uniform distribution is described here, other examples, such as regular distributions, can also be used to determine collection conditions. For instance, in the case of a regular distribution, after determining the number of data points required for each measurement range, wafer coordinates with high correlation to each measurement range are acquired and used as collection conditions to achieve that number of data points. In this way, collection conditions for achieving any distribution can be determined. Moreover, a statistic refers to a value used to evaluate correlation. For example, the aforementioned range of 2% to 98% is equivalent to a statistic (the same concept as the q-quantile mentioned above). Also, the statement above, "to determine the wafer coordinates to be collected in a way that ensures the same number of data points in each of the divided collection ranges," is a consideration for using a uniform distribution in machine learning. For example, if there are 100 measurements of m to 2m nanometers and only 10 measurements of 2m to 3m nanometers, the learning results will be more influenced by the m to 2m nanometer measurements than by the 2m to 3m nanometer measurements. Therefore, it is best to use data with a uniform distribution during learning (ideally, each measurement value should appear in the same number of occurrences) (ideally, each measurement value (each evaluation value) should have the same number). Therefore, dividing the collection range to achieve a uniform distribution is important. For example, in Figure 6, although the deviation values ​​at both ends are excluded, a large number of measurement values ​​still exist. In this case, data should be added and collected in a way that ensures the number of data points is the same, or a portion should not be used for learning.

[0037] Figure 10 is a diagram illustrating the correlation evaluation with device ID. In Figure 10, the vertical axis represents the values ​​used in the correlation evaluation. For example, if the evaluation value is a length measurement, the length measurement observed in each device will be different. In the case of Figure 10, if a high evaluation value is desired, the evaluation value (length measurement) is collected from device A, and if a low evaluation value is desired, the evaluation value (length measurement) is collected from device B. The device ID referred to here can be either the device ID related to the manufacturing device or the device ID related to the evaluation device.

[0038] As described above, Figures 9 and 10 illustrate the correlation evaluation when the wafer coordinates or device ID is used as the control possible parameter 701. By performing the correlation evaluation with the control possible parameter 701 in this way, the conditions for collection that should be performed to achieve the distribution of evaluation values ​​for the target can be identified. Furthermore, the control possible parameter 701 can also be a value other than the wafer coordinates or device ID. In this invention, the control possible parameter can be used when determining the collection conditions. Also, the value used for correlation evaluation can also be other than the measurement value. For example, the defect type or defect number can also be used. In this case, wafer coordinates or devices that are prone to specific defect types can be identified, and data collection can be performed in a focused manner. Also, the evaluation value related to the image can also be used. Furthermore, Figures 9 and 10 illustrate an example of determining the collection conditions using a single control possible parameter, but a combination of multiple control possible parameters can also be used to evaluate correlation. For example, correlation can be evaluated according to each combination of device and wafer coordinates. Furthermore, although it is described that the collection conditions are determined using controllable parameters with high correlation, they can also be set as collection conditions independent of correlation. For example, conditions for collecting data from the entire wafer coordinate system can be set independently of correlation, or collection conditions can be set in a manner that includes multiple device IDs.

[0039] If the above is summarized, the collection conditions are determined through the following procedure. First, the requirements regarding the collection scope are determined. This refers to the collection scope or the number of target data to be collected within each of the divided collection scopes.

[0040] Next, for the collection range or each of the divided collection ranges, a control parameter with a high correlation evaluation value is specified. This is, for example, when the collection range is related to the measurement value, by specifying a control parameter that indicates the possibility of observing the divided collection range, i.e., the range of the divided measurement value.

[0041] Next, highly relevant control parameters specific to the collection range or each of the divided collection ranges are selected and added to the collection conditions. For example, when the collection range is related to the measurement value, control parameters that indicate the possibility of observing the divided collection range, i.e., the range of the divided measurement values, are added to the collection conditions. More than one control parameter can be specified for the collection range or each of the divided collection ranges.

[0042] Finally, collection conditions can also be added according to pre-set conditions. That is, wafer coordinates can be added to the collection conditions in a manner that contains a predetermined number of points or more, or they can be added to the collection conditions in a manner that contains multiple device IDs.

[0043] Based on the above, the collection conditions will include: possible control parameters that should be collected in the evaluation device in order to collect the number of data points required for the collection range or each of the divided collection ranges.

[0044] (iii) Step S803: The collection procedure generation unit 704 generates a collection procedure according to the collection conditions determined in step S802. The collection procedure referred to here is the procedure required for evaluating the product in the evaluation device 203 and collecting data (or information indicating where the data should be collected). This collection procedure is generated in a manner that includes the control possible parameters 701 set as collection conditions in step S802. That is, the collection procedure generation unit 704 can also create a collection procedure for the product or each process to be collected, containing control possible parameters 701 such as wafer coordinates or device IDs set through correlation evaluation, and then, regardless of the correlation evaluation, make the parameters such as wafer coordinates or device IDs meet the predetermined conditions. The predetermined conditions referred to here include wafer coordinates at a predetermined location, a predetermined number or more manufacturing devices, or evaluation devices, etc.

[0045] Furthermore, regarding the creation of the collection procedure, the collection procedure used in the initial data collection can be utilized, or new collection conditions can be created by editing the already set items. That is, it is possible to consider editing only the wafer coordinates to be captured and changing the device to be published in the existing collection procedure.

[0046] (iv) In step S804, the collection instruction unit 705 publishes the information of the collection procedure generated in step S803 to the evaluation device 203 (the publishing target can be specified by the operator). At this time, if collection conditions involving multiple manufacturing devices or evaluation devices are set in step S802, the collection instruction unit 705 instructs that data from the set manufacturing devices / or evaluation devices can be collected. That is, when collecting data from products processed in multiple manufacturing devices, the collection instruction unit 705 can also publish the ID of the manufacturing device and the collection object associated with the collection conditions to multiple evaluation devices. Data is collected when a product such as a wafer associated with the ID of the manufacturing device is fed into the evaluation device. If collection conditions involving data from multiple evaluation devices are set, the collection procedure and collection object can also be published to the evaluation device, causing it to collect data when it observes the data of the collection object.

[0047] Furthermore, the data collection completion determination unit 207 determines whether data collection has been completed according to the predetermined criteria within the collection range determined by the data collection range determination unit 205. This is, for example, a process that divides the collection range into several smaller ranges and determines whether a predetermined amount of data has been collected for each smaller range. Here, the predetermined amount for each smaller range means a amount predetermined in order to achieve a target distribution shape such as uniform or regular distribution throughout the entire collection range. Then, if at least the predetermined amount of data (evaluation value) has been successfully collected in each smaller range, the data output from the evaluation device 203 corresponding to the successfully collected data and the data obtained by applying the output data to the learning model can be used to learn (relearn) the learning model.

[0048] Furthermore, it can be designed so that even if not included in the collection conditions, it is still possible to evaluate whether the wafer coordinates or device ID, etc., meet the pre-defined criteria, and only if they are met is it considered complete. With such a design, the learning dataset can be constructed in a way that satisfies the defined conditions. For example, it is also possible to consider situations where insufficient data can be collected in the edge area of ​​the wafer. In such cases, if data is collected from the target device and the operator determines that it is permissible, then the data can be used directly even if the learning data is slightly insufficient.

[0049] Furthermore, the collection completion determination caused by the collection completion determination unit 207 and the collection scope determination caused by the collection scope determination unit 205 can be implemented interactively. This allows the collection scope to be modified (e.g., expanded) by adding collected data during the collection of data within the initially determined collection scope. Furthermore, in cases where collection is difficult, the operator can be notified of these difficult conditions. The operator can then consider creating and adding the data showing the difficult collection conditions to the learning data. For example, if data for a specific length value is insufficient, the manufacturing conditions can be changed to allow the collection of that length value. Additionally, a display unit can be provided to show the operator the data collection status (see the collection condition display unit 1102 or collection procedure display unit 1_1103 in Figure 11). This allows the operator to confirm the data collection status and determine whether the learning process is successful.

[0050] <Example of GUI Configuration> Figure 11 is an illustration of an example of the configuration of an operator confirmation screen (GUI: Graphical User Interface) 1100. The operator confirmation screen 1100 includes: an object display unit 1101, a collection condition display unit 1102, a collection process display unit 1_1103, a collection process display unit 2_1104, a collection process confirmation unit 1105, and a collection execution instruction unit 1106.

[0051] The object display unit 1101 is a field used to display information about the machine learning model of the learning object or information about the evaluation object of the learning data.

[0052] The collection condition display unit 1102 is used to display the collection range determined by the collection range determination unit 205 or the collection conditions determined by the collection condition determination unit 703. In this way, the operator can confirm the necessary collection range or the sufficiency of the collection range.

[0053] Collection procedure display units 1_1103 and 2_1104 are display areas for confirming information required for the proposed collection procedure. Collection procedure display unit 1_1103 displays the collection procedure related to wafer coordinates. It can also be displayed together with statistics for each wafer coordinate based on the proposed collection wafer coordinates. Collection procedure display unit 2_1104 displays the collection procedure related to device IDs, highlighting the proposed collection device IDs. It can also be displayed here together with statistics for each device ID as the basis.

[0054] The collection procedure confirmation unit 1105 is a display area for a button that can be pressed when the collection procedure is confirmed. Once the operator presses the button when they want to confirm the collection procedure, the completed collection procedure is displayed, prompting the operator to confirm it.

[0055] The collection execution instruction unit 1106 is a display area for a button that can be pressed during the execution of data collection. Once the operator presses the button to instruct the execution of data collection, information on the collection procedure will be published to each evaluation device, and data collection will be performed in the device to which the publication is intended.

[0056] <Modification> The above embodiments describe the data collection required for learning a specific model. However, it is also possible to learn multiple models simultaneously. That is, this is an example of learning models that differ depending on each manufacturing process or evaluation condition. In such cases, the actions of the data collection condition-specific unit 204 can be modified using information about the learning object of the machine learning model. Specifically, changes such as modifying the conditions of the data extracted in the evaluation object extraction unit 402, or modifying the collection program generated in the collection program generation unit 704 to control the applicable object using information about the learning object, and implementing changes according to each machine learning model for the collection completion determination unit 207, can be considered.

[0057] Furthermore, the operator may also use a method that specifically identifies the insufficient data during the relearning process and collects only the insufficient data, or may discard the existing data and collect new data. In cases where the image quality of the evaluation image changes significantly due to changes in product materials, etc., it is possible that past data may hinder learning. In such cases, simply collecting the insufficient data will not result in a well-performing machine learning model. Therefore, all data may be collected again. Furthermore, to automate these judgments, a mechanism may be introduced that evaluates changes in the image quality of the evaluation image and, upon reaching a certain level of change, collects all data again.

[0058] Furthermore, the information necessary for the determination of the machine learning model of the object can be stored in the memory unit 202 and used for action control when collecting each machine learning model.

[0059] <Effects of this embodiment> According to the embodiment of the present invention, the data necessary for the learning of a machine learning model can be collected efficiently, and a highly reliable model can be constructed. Furthermore, according to the embodiment of the present invention, when the learning of a machine learning model is necessary, the collection conditions and procedures required for collecting the learning dataset can be formulated.

[0060] Here, "collection conditions" means the values ​​of data that the learning dataset should collect in order to learn a highly reliable machine learning model. If the learning dataset is a quantitative dataset collected for each value, it is expected that the attributes of the data input into the machine learning model will be included, which can prevent the machine learning model from making unexpected outputs. Furthermore, "collection procedure" means the procedure required for data collection to achieve the above-mentioned collection conditions (e.g., a formula indicating from which location to collect data). This is equivalent to data that records the parameters required to control the operation of the evaluation device. For example, for each collected object that the machine learning model is learning from, a procedure for acquiring evaluation images is recorded. This procedure may include information about the product being evaluated, such as imaging conditions or wafer coordinates.

[0061] These collection conditions or procedures can also prompt the operator to confirm, or can be automatically issued to the device for collection when the specified conditions are met.

[0062] Furthermore, according to the configuration of the present invention, the operator can efficiently manage the quality of learning data for machine learning models. By checking the screen example of the present invention, the operator can confirm the attributes and value range of the data to be collected, confirm the current collection status, and by checking the proposed collection procedure, confirm the collection status of the learning dataset. Even if there is insufficient data, including data that is difficult to collect, the operator can perform intentional operations to generate insufficient data and collect the learning data. This is equivalent to, for example, changing the design information when there is insufficient data for a specific measurement value, so that the insufficient measurement value data will be generated.

[0063] <Other> This invention is not limited to the embodiments described above and may include various modifications. For example, the embodiments described above are provided in detail to aid in understanding the invention, and the invention is not limited to having all the described components. Furthermore, a portion of a certain component may be replaced with another component, and other components may be added to a component. Moreover, for a portion of a component, other components may be added / deleted / replaced.

[0064] Furthermore, the functions of the present embodiment described above can also be implemented using software code. In this case, a memory medium containing the code is provided to a system or device, and the computer (or CPU or MPU) of the system or device reads the code stored in the memory medium. In this case, the functions of the aforementioned embodiment are implemented from the code itself read from the memory medium, and the code itself and the memory medium storing it constitute the present invention. As the memory medium required to supply such code, examples such as floppy disks, CD-ROMs, DVD-ROMs, hard disks, optical discs, optical magnetic disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc., can be used.

[0065] Alternatively, based on the instructions in the program code, part or all of the actual processing can be performed by the OS (operating system) or the like running on the computer, thereby realizing the functions of the aforementioned implementation. Alternatively, after the program code read from the memory medium is written into the computer's memory, based on the instructions in the program code, part or all of the actual processing can be performed by the computer's CPU or the like, thereby realizing the functions of the aforementioned implementation.

[0066] Alternatively, the program code of the software that implements the functions of this embodiment can be transmitted via a network and stored in a memory device such as a hard drive or memory of the system or device, or in a memory medium such as a CD-RW or CD-R. When in use, the computer (or CPU or MPU) of the system or device can read out and execute the program code stored in the memory device or memory medium.

[0067] The processing procedures and techniques described herein are not inherently associated with any particular device and can be implemented through combinations of components. Furthermore, various types of general-purpose elements can be added. Dedicated devices can also be constructed to perform the functions of this embodiment. Moreover, various functions can be formed by appropriately combining the plurality of constituent elements disclosed in this embodiment. For example, several constituent elements can be deleted from all the constituent elements shown in the embodiments and various examples, and constituent elements can be appropriately combined across different embodiments.

[0068] While specific embodiments are described in this specification, they are not intended to be limiting in any way, but rather are for illustrative purposes (to help understand the invention). Anyone skilled in the art will understand that there are many combinations of hardware, software, and firmware suitable for carrying out the invention. For example, the software can be implemented using a wide range of programming or scripting languages, such as assembly languages, C / C++, Perl, Shell, PHP, Java (registered trademark), etc.

[0069] Furthermore, in the above-described embodiments, control lines or information lines are indicated only for the sake of illustration, and not all control lines or information lines may necessarily be present on the product. All components may also be interconnected.

[0070] Furthermore, any person skilled in the art will be able to understand other embodiments of the present invention from an examination of these embodiments. The description and specific examples are merely typical examples, and the scope and spirit of the present invention are disclosed in the following claims. [Simplified Explanation of the Diagram]

[0009] [Figure 1] A schematic diagram of the variation of the measured value distribution as envisioned in this invention. [Figure 2] A schematic diagram of an example of the structure of this invention. [Figure 3] A schematic diagram of the structure of the memory unit. [Figure 4] A schematic diagram of an example of the structure of the collection range determination unit. [Figure 5] A flowchart of the collection range determination unit. [Figure 6] A schematic diagram of the statistical calculation unit. [Figure 7] A schematic diagram of an example of the structure of the collection method indication unit. [Figure 8] A flowchart of the collection method indication unit. [Figure 9] A schematic diagram of the correlation evaluation using wafer coordinates. [Figure 10] A schematic diagram of the correlation evaluation using device ID. [Figure 11] A schematic diagram of a screen example of this invention.

Claims

1. A data processing system comprising a computer system having at least one evaluation device and a system for applying first information output from the evaluation device to a learning model to output second information, wherein, The aforementioned computer system performs the following processes: calculating an evaluation value, i.e., the third information, from at least one of the aforementioned first information or the aforementioned second information; performing statistical calculations on a plurality of the aforementioned third information to calculate a statistical quantity, and determining the collection scope for collecting the aforementioned third information based on the statistical quantity; collecting data from the plurality of the aforementioned third information calculated based on the output of at least one of the aforementioned evaluation devices, that is included in the aforementioned collection scope of the aforementioned third information; and relearning the aforementioned learning model using the previously collected aforementioned third information, and the aforementioned first information and the aforementioned second information corresponding to the previously collected aforementioned third information.

2. The data processing system as described in claim 1, wherein, The pre-record computer system divides the pre-record collection range into multiple smaller ranges. If a predetermined number of pre-record third information can be collected in each smaller range, the pre-record first information and pre-record second information corresponding to the collected pre-record third information are used to relearn the pre-record learning model.

3. The data processing system as described in claim 1, wherein, The computer system described above performs the following: processing of the correlation evaluation of the parameters used in the evaluation of at least one evaluation device described above, which are specified parameters, with the third information described above; and processing of determining the collection conditions of the data included in the collection scope described above in a manner that can achieve the specified target distribution using the results of the correlation evaluation described above.

4. The data processing system as described in claim 2, wherein, The computer system described above performs the following: processing of the correlation evaluation of the parameters used in the evaluation of at least one evaluation device described above, which are specified parameters, with the third information described above; and processing of using the results of the correlation evaluation described above to determine the collection conditions of the data included in the collection scope or a smaller scope described above in a manner that can achieve the specified target distribution.

5. The data processing system as described in claim 4, wherein, The parameters for the multiple categories are specified; the computer system determines the collection conditions based on the parameters for each of the multiple categories, in a way that a certain number of data are included in the collection scope or a smaller scope.

6. The data processing system as described in claim 4, wherein, The aforementioned computer system performs the following processes: generating a data collection procedure in accordance with the aforementioned collection conditions, in a manner that includes the aforementioned parameters that have been set as the aforementioned collection conditions; publishing the aforementioned data collection procedure to at least one evaluation device; and collecting the aforementioned first information obtained in accordance with the published collection procedure, and the aforementioned second and third information related to the aforementioned first information obtained in accordance with the published collection procedure.

7. The data processing system as described in claim 6, wherein, The foregoing at least one evaluation device is for products processed by at least one manufacturing device included in the foregoing parameters, and will collect the foregoing first information obtained in accordance with the foregoing collection procedure that has been published, and the foregoing second information and foregoing third information related to the foregoing first information obtained in accordance with the foregoing collection procedure that has been published.

8. The data processing system as described in claim 7, wherein, The preceding computer system performs the following: It displays on the screen whether the predetermined amount of data can be collected within the preceding collection range or within a smaller preceding range.

9. A data processing method, using a computer, for learning a learning model that applies first information output from at least one evaluation device and outputs second information, comprising: calculating an evaluation value, i.e., third information, from at least one of the first or second prior information; calculating a statistic by performing statistical calculations on a plurality of the third prior information, and determining a collection range for collecting the third prior information based on the statistic; collecting data from the plurality of third prior information calculated based on the output of at least one evaluation device, that is included in the collection range of the third prior information; and relearning the prior learning model using the collected third prior information and the corresponding first and second prior information.

10. The data processing method described in claim 9, wherein, The pre-record computer divides the pre-record collection range into multiple smaller ranges. If a predetermined number of pre-record third information can be collected in each smaller range, the pre-record first information and pre-record second information corresponding to the pre-record third information are used to relearn the pre-record learning model.

11. The data processing method described in claim 9, wherein, It also includes: the parameters used in the evaluation of at least one evaluation device in the preceding period, which are specifiable parameters, and the correlation evaluation with the third information in the preceding period; and the determination of the collection conditions of the data included in the collection scope of the preceding period in a manner that can achieve the specified target distribution using the results of the correlation evaluation in the preceding period.

12. The data processing method as described in Request 10, wherein, It also includes: the parameters used in the evaluation of at least one evaluation device in the preceding period, which are specifiable parameters, and the correlation evaluation with the third information in the preceding period; and the determination of the collection conditions of the data included in the collection scope or the smaller scope of the preceding period in a manner that can achieve the specified target distribution using the results of the correlation evaluation in the preceding period.

Images