Data manipulation device and data manipulation method

Abstract

A data manipulation device that manages and processes data used in a machine learning model that performs predetermined processing on image information of a manufactured product, the data manipulation device being configured to include: an identification information imparting unit that imparts identification information to the image information, the identification information identifies whether data processing related to a machine learning model can be applied; a storage unit that associates and stores the identification information and attribute information that specifies various conditions when acquiring the image data that may be a factor of variation in the use data that affects the prediction performance of the machine learning model; an analysis condition verification unit that verifies whether an analysis condition, which is a condition for performing analysis on the machine learning model, is satisfied; an attribute information acquisition unit differentiated by identification information that acquires, from the storage unit, the image information and the attribute information associated with the identification information corresponding to the analysis condition when the analysis condition is satisfied; and a statistic calculation unit that calculates a statistic to be used in the management of the machine learning model on the basis of the acquired image information and attribute information.

Description

TECHNICAL FIELD

[0001] The present application relates to a data operation device that performs data management and data processing in a machine learning model. BACKGROUND

[0002] The performance of a machine learning model depends on various data such as learning data, processing target data, and the like. Therefore, managing the distribution and fluctuation of the learning data and the processing target data is important in managing the performance of the machine learning model. For example, in Patent Literature 1, it is described that a statistical amount of learning data and processing target data is monitored and concept drift is detected, and when concept drift is detected, a user is notified and prompted to learn again.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: WO2022 / 196666A1 SUMMARY OF THE INVENTION

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In Patent Literature 1, a method is described in which, when a machine learning model has generated concept drift as an application target, the machine learning model is managed by relearning using data that has generated concept drift and the like.

[0008] However, in the case where an abnormality has occurred due to an abnormality of a manufacturing device, an inspection device, or the like, relearning using data that has generated concept drift cannot solve the abnormality of the device. Therefore, in the present application, a method is provided for analyzing whether the cause of a change in the recognition performance of a machine learning model is a cause due to an unlearned data distribution or a cause due to a change in the state of a specific device itself and appropriately coping with it. In addition, a method is provided for easily managing a machine learning model by analyzing factors that degrade the prediction / inference performance of the machine learning model by determining data that should be included in learning data.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] To solve the above problems, the present application is a data operation device that performs management and processing of utilization data used in a machine learning model that performs predetermined processing on image information of manufactured products, the data operation device having: an identification information assigning section that assigns identification information to the image information, the identification information identifying whether or not data processing related to the machine learning model can be applied; a storage section that stores the identification information in association with attribute information that specifies various conditions at the time of acquisition of the image information that can be a fluctuation factor of the utilization data that affects the prediction performance of the machine learning model; an analysis condition verification section that verifies whether or not an analysis condition that is a condition for performing analysis on the machine learning model is satisfied; an attribute information acquisition section that acquires, when the analysis condition is satisfied, the image information and the attribute information associated with the identification information corresponding to the analysis condition from the storage section, distinguished by the identification information; and a statistical quantity calculation section that calculates a statistical quantity used in the management of the machine learning model, based on the acquired image information and attribute information.

[0011] Effects of Invention

[0012] According to the present application, the management of the machine learning model can be performed more reliably.

[0013] The above problems, structures, and effects are made clear by the following embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a diagram showing an outline of a production line in an embodiment of the present application.

[0015] Figure 2 is a diagram showing a structural example of a machine learning model in an embodiment of the present application.

[0016] Figure 3 is a diagram showing the structure of a data operation device in an embodiment of the present application.

[0017] Figure 4 is a flowchart showing a processing flow in an embodiment of the present application.

[0018] Figure 5 is a diagram showing the structure of a data operation device in a second embodiment of the present application.

[0019] Figure 6 is a diagram showing a screen structural example in a second embodiment of the present application.

[0020] Figure 7A is a diagram showing an example of a statistical quantity in a second embodiment of the present application.

[0021] Figure 7B is a diagram showing an example of a statistical quantity in a second embodiment of the present application.

[0022] Figure 7C This is a diagram showing an example of statistics in the second embodiment of the present invention.

[0023] Figure 7D This is a diagram showing an example of statistics in the second embodiment of the present invention.

[0024] Figure 7E This is a diagram showing an example of statistics in the second embodiment of the present invention.

[0025] Figure 7F This is a diagram showing an example of statistics in the second embodiment of the present invention.

[0026] Figure 8 This is a flowchart showing the flow of analyzing the cause and formulating a countermeasure plan when performance deteriorates in the second embodiment of the present invention.

[0027] Figure 9 It is a diagram showing the structure of a data operation device in a third embodiment of the present invention.

[0028] Figure 10A This is a diagram showing an example of statistics in the third embodiment of the present invention.

[0029] Figure 10B This is a diagram showing an example of statistics in the third embodiment of the present invention.

[0030] Figure 10C This is a diagram showing an example of statistics in the third embodiment of the present invention.

[0031] Figure 10D This is a diagram showing an example of statistics in the third embodiment of the present invention.

[0032] Figure 11 It is a diagram showing the structure of a data operation device in a fourth embodiment of the present invention.

[0033] Figure 12A This is a diagram showing an example of statistics in the fourth embodiment of the present invention.

[0034] Figure 12B This is a diagram showing an example of statistics in the fourth embodiment of the present invention.

[0035] Figure 12C This is a diagram showing an example of statistics in the fourth embodiment of the present invention.

[0036] Figure 12D This is a diagram showing an example of statistics in the fourth embodiment of the present invention.

[0037] Figure 13is a drawing showing an example of a data table of a storage section in an embodiment of the present application. DETAILED DESCRIPTION

[0038] Embodiments of the present application will be described below with reference to the drawings. Note that detailed description of portions common to each embodiment is omitted.

[0039] Embodiment 1

[0040] In this embodiment, a production line for manufacturing a product such as a semiconductor is assumed. Figure 1 A schematic of the assumed production line of this embodiment is shown in FIG. 1. In the production line assumed in this embodiment, a product 102 is processed by a manufacturing device 101, and whether or not the product can be processed as assumed is evaluated by an evaluation device 103. Here, for simplicity, only one manufacturing process of one manufacturing device is described, but a plurality of manufacturing processes in which a plurality of manufacturing devices are connected in series can also be the target.

[0041] Note that a plurality of manufacturing devices 101 and a plurality of evaluation devices 103 can also exist in the same process. For example, in the case of a production line for a semiconductor, a plurality of devices that perform the same processing, such as an etcher and an exposure device, exist.

[0042] The evaluation device 103 is, for example, an inspection and measurement device that uses an optical inspection device or a scanning electron microscope (SEM), and a plurality of devices that perform the same inspection or measurement processing exist. Note that the evaluation device 103 has a calculation processing section in addition to the functions of the evaluation device itself, and can perform prediction processing and inspection and measurement processing by installing a machine learning model as described later.

[0043] The product 102 can be a finished product or a state halfway through the production line.

[0044] The computer 104 acquires various data related to inspection and measurement from the evaluation device 103 and stores the data in a storage section 303. Note that the computer 104 performs processing such as learning and evaluation of a machine learning model using data stored in the storage section 303 and stores the results in the storage section 303. Note that in the case where annotation and the like are required when learning and evaluating the machine learning model, a structure in which the results of the annotation are stored as training information in the storage section can also be added.

[0045] Further, the computer 104 can distribute the machine learning model obtained through the learning processing to the evaluation device 103, and the evaluation device 103 can perform the prediction processing, the inspection measurement processing using the distributed machine learning model. Alternatively, the machine learning model can be installed in the operation processing unit of the manufacturing device 101 to be executed. Alternatively, the machine learning model can be executed not only by the operation processing units of the manufacturing device 101 and the evaluation device 103 but also by another computer (not illustrated) different from the computer 104.

[0046] Figure 2 The machine learning model 202 is the machine learning model in the present embodiment, and as described above, can be installed in the operation processing device in the manufacturing device 101 and the evaluation device 103, the computer 104, or another computer (not illustrated), but in the present embodiment, is described as being installed in the operation processing device in the evaluation device 103.

[0047] The machine learning model 202 receives the input data 201, performs the inference processing defined by the machine learning model 202, and outputs the output data 203. Here, the input data is data required according to the processing content of the machine learning model 202, and is, for example, an inspection measurement image obtained by optical inspection or SEM in the evaluation device 103, an acquisition condition of the image, design information of an inspection measurement object, or the like.

[0048] The output data 203 is prediction data or the like output according to the content of the machine learning model 202, and is, for example, an inference processing result including a prediction of a detection result of a defect, a foreign matter, or the like of a semiconductor wafer, a generation result of a reference image used in reference image comparison inspection of comparing an inspection image and a reference image, a noise removal prediction result of removing noise included in an inspection measurement image output from the SEM or the like of the evaluation device 103, an image emphasis prediction result of emphasizing a specific image region in the inspection measurement image, or the like.

[0049] However, in microfabrication such as a semiconductor production line, the state of the device easily affects the processing result of each process in the microfabrication. Therefore, the state of the manufacturing device 101 and the evaluation device 103 is monitored and appropriate maintenance work is performed. If such a production line is executed with the inference processing based on the machine learning model 202 or the like, when the processing condition of the manufacturing device 101 and the evaluation device 103 is changed, or when the state of a specific manufacturing device 101 and evaluation device 103 is changed, it is possible that data deviating from the data distribution learned from the machine learning model 202 is provided as the input data 201.

[0050] Typically, the inference performance of the machine learning model 202 may deteriorate due to input data 201 that deviates from the learning data distribution. Therefore, when the machine learning model 202 is used, a monitoring method suitable for each machine learning model 202 is used to confirm whether the performance has deteriorated.

[0051] For example, in applications where accurate predicted target images can be obtained with minimal effort, such as by removing noise from inspection-measured images, degradation of prediction performance can be monitored using values ​​such as image correlation between the predicted image and the predicted target image based on the machine learning model, or by checking the difference between measurement processing results. Alternatively, in defect inspection applications, humans can annotate random images to confirm consistency with the predicted results, or to check for changes in the distribution of detected defects or defect types. In measurement processing, changes in the distribution of measured values ​​can be checked to determine whether the output values ​​deviate from the assumed distribution.

[0052] This embodiment provides a method for coping with such situations and appropriately managing and operating the machine learning model 202. As will be described later, the inference performance of the machine learning model 202 is affected by changes in the input data 201 from the time of learning to the time of inference processing. Therefore, by managing the changes in the input data 201 for each factor of change, the performance management of the machine learning model 202 is facilitated.

[0053] Here, the following factors are considered as variable factors. The variable factors in manufacturing apparatus 101 are manufacturing conditions including the manufacturing apparatus ID representing the individual manufacturing apparatus and process conditions. Process conditions include focus value and dose in the case of an exposure apparatus, and gas flow rate and voltage in the case of an etcher.

[0054] In the case of a semiconductor production line, for example, factors of variation in the product 102 include wafer defects, the amount of foreign matter mixed in, coordinate information on the wafer, or circuit design information.

[0055] The variable factors in the evaluation device 103 are evaluation conditions including the evaluation device ID indicating the individual evaluation device and evaluation conditions. Here, the evaluation conditions are values ​​related to imaging conditions or acquisition conditions such as the acceleration voltage and probe current of the SEM.

[0056] In this embodiment, variable factors dependent on the machine learning model 202 are also managed. These variable factors include the model ID used to identify the applied machine learning model 202, measurement values ​​such as the dimensions of the measured object in measurement processing, and the number and type of defects detected in inspection processing. These factors may also include whether the circuit shape is included in the learning data.

[0057] In this way, as variable factors, there are various values ​​and conditions related to the manufacturing device 101, product 102, evaluation device 103, and machine learning model 202 that may affect the evaluation of the evaluation device 103 using the machine learning model 202 in the production line, and the information related to them is called attribute information.

[0058] Figure 3 and Figure 4 The structure of the data operation device for data management and data processing in this embodiment and its processing flow chart are shown. Figure 3 The parts shown can be Figure 1 It can be implemented by application software in the computer 104, or it can be implemented by dedicated hardware.

[0059] The storage unit 303 stores data 301. Data 301 is, for example, an inspection and measurement image acquired by the evaluation device 103 and is any of the input data 201 or output data 203 of the machine learning model 202 or training data for the machine learning model 202. Furthermore, the data is stored in the storage unit 303 in association with the aforementioned attribute information.

[0060] The identification information assigning unit 302 assigns identification information to each piece of data 301 stored in the storage unit 303. Here, identification information refers to information assigned to each management item, such as learning, evaluation, and monitoring of a machine learning model. This information is used to identify whether data processing, described later, can be applied. Specifically, this information includes information such as whether a data item is used for learning a specific model, whether it is used for evaluating a specific model, whether it is used for monitoring a specific model, whether performance has been determined to be degraded by monitoring, and a model ID that identifies the model.

[0061] Specifically, data 301 (image data) is stored in storage unit 303 in association with identification information and attribute information assigned by identification information assigning unit 302. Attribute information includes, among other things, the evaluation device ID as a variable factor related to the evaluation device, acquisition conditions such as the imaging conditions of input data 201, characteristic conditions such as measurement values ​​and defect types related to input data 201 as variable factors dependent on machine learning model 202, manufacturing conditions such as the manufacturing device ID and processing conditions as a variable factor related to the manufacturing device, and product conditions as a variable factor related to the product. Acquisition conditions may also include the imaging date and time.

[0062] In addition, it can also be saved in association with machine learning model information that is information related to learning or evaluation of the machine learning model 202 as needed. The machine learning model information is, for example, whether it is used for learning of the machine learning model 202, or used for evaluation, training information in the case of using supervised learning, information related to the machine learning model used, and the like. Figure 13 A data table example of the data 301, the attribute information 301B, and the identification information 301C saved in the storage 303 is shown.

[0063] Next, the management of the machine learning model 202 is described using the flowchart of Fig. 13. Figure 4

[0064] At the timing at which management of the machine learning model 202 is needed, the process shown in the flowchart of Fig. 14 is executed. Here, the timing at which management is needed refers to a timing at which a specific process is instructed, a predetermined time elapses, and the like. For example, in the case of performance monitoring of the machine learning model (Embodiment 2), it is a timing at which verification of performance degradation is instructed, a timing at which performance degradation is detected, in the case of verification of learning data before learning of the machine learning model (Embodiment 3), it is a timing at which the learning process is instructed, and in the case of performance verification of the model after learning of the machine learning model (model generation) (Embodiment 4), it is a timing at which the performance verification is instructed. Figure 4

[0065] In step S401, the analysis condition verification section 304 verifies whether or not the analysis condition is satisfied. Here, the analysis condition refers to a process that instructs detection of performance degradation, learning, or verification, and the like, and corresponds to the above-mentioned timing at which management is needed. That is, it is confirmed whether or not a trigger (opportunity) for verification of the data 301 is generated, and the content thereof (performance degradation detection, learning, verification). In the case where the analysis condition is satisfied, the process proceeds to step S402, and if not, the process ends.

[0066] In step S402, the attribute information acquisition section 305 that distinguishes by identification information determines, from the storage 303, the identification information that is determined by the analysis condition verification section 304 to satisfy the analysis condition, and acquires the data 301 associated with the identification information and the associated attribute information from the storage 303.

[0067] For example, in the case where it is determined that the analysis condition is satisfied by the learning process that instructs "machine learning model X (model ID = X)", in the example of the data of the storage 303 shown in Fig. 13, the image data 301 and the attribute information 301B (1300) in which "model ID: model X, learning data" is recorded as the identification information 301 of "#1" are acquired. Figure 13

[0068] ​​​In step S403, the statistical quantity calculation section 306 calculates various statistical quantities (specific examples of the statistical quantities will be described later in Embodiment 2) that are decided in advance for evaluating the inference performance of the machine learning model 202, based on the data and the attribute information thereof acquired by the attribute information acquisition section 305 distinguished by the identification information. After the calculation of the statistical quantities is completed, Figure 4 The flowchart illustrated in FIG. 6 ends.

[0069] According to the statistical quantities calculated here, the operator can make decisions related to the management of the machine learning model 202. Here, the management of the machine learning model 202 refers to the cause analysis at the time of performance degradation, the sufficiency determination of the learning data at the time of learning start, the model verification at the time of learning end, and the like, or various measures for maintaining the performance of the machine learning model 202 based on these analyses, determinations, verifications, and the like.

[0070] For example, as described above, the input data 201 is affected by the device states of the manufacturing device 101, the evaluation device 103, and the like, and thus if a maintenance work of a specific device is performed, the performance of the machine learning model 202 can sometimes be maintained. In addition, in a case where the change of the processing conditions of the manufacturing device 101 or the like is not due to a specific device, re-learning of the machine learning model 202 is required. In this way, according to the cause of the performance degradation, appropriate countermeasures such as re-learning and maintenance work can be implemented.

[0071] As described above, according to the present embodiment, the data 301 related to the machine learning model 202 is managed in association with the attribute information indicating the fluctuation factors thereof, and by the identification information associated with each category of the machine learning model 202, the acquisition becomes easy, and the operator can provide the information required for the management of the machine learning model 202.

[0072] Embodiment 2

[0073] In the present embodiment, the cause analysis at the time of performance degradation of the machine learning model 202 is described. Figure 5 The structure of the present embodiment related to the cause analysis at the time of performance degradation is described. Figure 5 Each section illustrated in FIG. 6 can be realized by the application software within the computer 104 as in Embodiment 1, or can be realized by a dedicated hardware, respectively. The structure of the present embodiment is different from that of Embodiment 1 in that the correlation degree evaluation section 501 and the output section 502 are added. Figure 1 Figure 3 The correlation degree evaluation section 501 evaluates the correlation degree with the performance degradation for each attribute information according to the statistical quantities calculated by the statistical quantity calculation section 306, and formulates countermeasures for maintaining the inference performance based on the evaluation result.

[0074] That is, the correlation degree evaluation section 501 evaluates the correlation degree with the performance degradation for each attribute information according to the statistical quantities calculated by the statistical quantity calculation section 306, and formulates countermeasures for maintaining the inference performance based on the evaluation result. ​

[0075] The output unit 502 presents information related to cause analysis and measures for maintaining inference performance to the operator based on the aforementioned degree of correlation, and is implemented through a display device or the like.

[0076] When analyzing the cause of performance degradation, the identification information of whether the data 301 is detected as having performance degradation is used among the identification information added by the identification information adding unit 302 .

[0077] The analysis condition verification unit 304 verifies whether a performance degradation cause analysis is necessary. This verifies whether the number of performance degradation detections within a predetermined time interval or a predetermined number of inference processing cycles of the machine learning model 202 exceeds a predetermined threshold. Examples include the number of performance degradation detections per unit time or the number of performance degradation detections after a predetermined number of inference processing cycles or monitoring processes. The operator sets these values, which are appropriate for the management of the machine learning model 202, and are determined based on, for example, empirical rules.

[0078] Detection of performance degradation is performed in the same manner as the model monitoring method described in Example 1. Specifically, when inspecting and measuring image noise removal, the image correlation between the predicted image and the predicted target image is evaluated, and the difference in measurement processing results is examined. When inspecting for defects, humans annotate random images and evaluate the prediction results and consistency, or assess changes in the distribution of the number of detected defects or defect types. When performing measurement processing, the detection is performed by confirming whether the distribution of measured values ​​has changed and whether the output values ​​deviate from the assumed distribution of measured values.

[0079] When performance degradation cause analysis is required under the above-mentioned conditions, analysis condition verification unit 304 initiates performance degradation cause analysis of the machine learning model and transmits the model ID for which performance degradation was detected to identification information-based attribute information acquisition unit 305. Identification information-based attribute information acquisition unit 305 then retrieves data associated with the transmitted model ID from storage unit 303. In this case, only data for which identification information was assigned by identification information assignment unit 302 indicating performance degradation has been detected may be acquired, or data for which performance degradation was evaluated but no degradation was detected may also be acquired.

[0080] The statistics calculation unit 306 calculates various statistics based on the attribute information and identification information acquired by the attribute information acquisition unit 305 classified by identification information. Specifically, based on the factors of performance degradation, the statistics are calculated as follows: (1) statistics related to evaluation device factors, (2) statistics related to product factors, (3) statistics related to manufacturing factors, (4) statistics related to machine learning model factors, and (5) statistics related to machine learning model factors for each device. The contents of each of these statistics are described in detail below.

[0081] (1) The statistic 701 related to the evaluation device factor is a statistic related to the variable factor based on the inherent reasons of the evaluation device. In other words, it is an indicator for confirming whether the predicted performance degradation occurs only in a specific evaluation device, such as Figure 7A As shown, the frequency of performance degradation detection is calculated for each evaluation device ID over a predetermined time period. In this example, the device with evaluation device ID "B" has observed significant performance degradation over time. Therefore, the operator can assess the likelihood of predicted performance degradation based on the unique state and conditions of evaluation device B.

[0082] Alternatively, the frequency of performance degradation detection for each predetermined time period may be evaluated for each evaluation condition, rather than for each evaluation device ID. The evaluation condition may be a single parameter such as acceleration voltage, or each combination of acceleration voltage and probe current.

[0083] This allows the operator to identify the evaluation device ID or evaluation conditions on which detection of performance degradation is concentrated.

[0084] (2) The statistics 702 related to product factors are statistics related to the variable factors based on the inherent reasons of the product. Figure 7B As shown, it is an indicator used to confirm whether performance degradation has occurred significantly in a specific part of the product. For example, 702A uses light and dark to indicate the detection frequency (detection ratio) of performance degradation at each position on the wafer product (the darker the color, the higher the detection frequency).

[0085] In addition to the locations where performance degradation is detected, the calculation can also use the locations where defects are detected during defect inspection (702B) and the locations where length measurement errors occur when measurement fails (702C). This allows the operator to perform evaluations such as adding specific locations to the learning data for relearning when performance degradation is concentrated.

[0086] These statistics can also be displayed together. In this case, if the statistical trends of performance degradation locations are similar to those of defect locations or length measurement errors, the operator can infer that the resulting defects or length measurement errors are the main factors of performance degradation and perform further learning based on the collected product status confirmation and learning data. Alternatively, as in the example of evaluating device factor 701, the frequency of performance degradation detection for each product ID can be used for evaluation.

[0087] (3) The manufacturing equipment factor 703 is a statistic related to the variable factor based on the inherent reasons of the manufacturing equipment. Figure 7CAs shown, the frequency of detection of performance degradation for each predetermined time width is calculated for each manufacturing apparatus ID. Here, the frequency of occurrence of performance degradation of the manufacturing apparatus c significantly changes over time compared to other manufacturing apparatuses, and thus the operator can evaluate that, as a cause of performance degradation, it is likely that there is a cause inherent to the manufacturing apparatus c, rather than being based on the prediction model.

[0088] Further, the frequency of detection of performance degradation for each predetermined time width can also be evaluated not for each manufacturing apparatus ID but for each manufacturing condition. For example, the manufacturing condition can be an individual parameter such as a focus value, or evaluation of each combination of a focus value and a dose and the like.

[0089] By the above, the operator can determine the manufacturing apparatus ID or manufacturing condition in which the detection of performance degradation is concentrated.

[0090] (4) The machine learning model factor 704 is a statistical quantity related to a variation factor that depends on the machine learning model 202. As shown in Figure 7D In the case where the distribution, shape of the data group used at the time of learning of the machine learning model varies over time at the time of evaluation, the degree of variation is calculated as an evaluation index. For example, in the distribution of measurement values, detected defect types, the information amount of the distribution during evaluation and the distribution for reference (learning) can be calculated, and in the example shown in Figure 7D In the example shown, it is known that, although there is no large difference in the distribution of data used at the time of learning and analysis in the apparatus B, the distribution of data used at the time of learning and the like in the apparatus A greatly varies over time.

[0091] Here, as the information amount of the evaluation index, the Kullback-Leibler information amount can be used, or the Wasserstein distance can be used. In addition to the distribution of measurement values, detected defect types, a distribution related to other feature quantities used by the machine learning model 202 can also be used. In addition, a statistical quantity other than the information amount can also be used.

[0092] Here, the evaluation period is a predetermined time width like the apparatus factor 701, the product factor 702, and the manufacturing apparatus factor 703, and the distribution for reference is a distribution referred to in order to evaluate the variation amount. For example, it is a distribution related to data used at the time of learning, or a distribution related to data at the time of evaluation in which performance degradation is not detected. With respect to data that is difficult to collect, such as abnormal data, data of rare phenomena, a distribution related to data collected separately from data at the time of learning can also be used.

[0093] For example, as Figure 7EAs shown, (1) in the case where the data distribution of the measurement values varies between the learning time and the analysis time (machine learning model measurement factor 705), a histogram of the measurement values is calculated based on the predetermined intervals and interval widths, the information amount between the learning time and the analysis time is calculated, and, (2) in the case where the distribution of the detected defect types varies between the learning time and the analysis time (machine learning model inspection factor 706), the classified distribution of the detected defect types is calculated, the information amount between the learning time and the analysis time is calculated, and, thereby, the Figure 7D the time variation of the evaluation index.

[0094] As described above, in the machine learning model factor 704, the information amount from the reference distribution is calculated and evaluated for each predetermined time width with respect to the distribution of the measurement values or the detected defect types. If the evaluation is based on the information amount, it is possible to detect even in the case where the values deviating from the reference distribution are included, that is, even in the case where the specific values are extremely biased within the range of the reference distribution. The operator analyzes the problem caused by the information amount.

[0095] For example, the information amount can be confirmed for each evaluation device, product, manufacturing device, and each variation factor of the machine learning model, and the variation factor capable of identifying the performance degradation factor can be determined. In addition, the detection ratio of the performance degradation can be used in addition to the detection frequency.

[0096] (5) The device-by-device machine learning model factor 707 is an index for confirming which one of the plurality of models operating in a specific device does not have performance degradation, Figure 7F is an example of calculating the detection frequency of the predicted performance degradation of the learning models a to d operating in the device A as a statistic. In such an example, the operator compares the detection frequency of each model, and, in the case where the performance degradation is detected in a large amount in only a specific model (in the case where the model b corresponds to the model b in Figure 7F , it is possible to promote analysis as a phenomenon occurring in the model.

[0097] Further, in the case where the detection frequency of the performance degradation does not greatly deviate among many models and the detection frequency in a specific device is more than that in other devices, it is possible to promote analysis as a possibility that some kind of problem has occurred in the device itself.

[0098] Figure 6 A screen structure example of the output section 502 is shown. The screen of the output section 502 in the cause analysis at the time of performance degradation is constituted by a report target display section 601, a report condition display section 602, a correlation degree evaluation result display section 603, and a determination basis display section 604.

[0099] The report target display section 601 displays information related to the machine learning model 202 in which performance degradation is detected. This is a machine learning model ID or information related to the learning conditions of the machine learning model.

[0100] The report condition display section 602 displays the condition in which performance degradation is detected. This is a verification condition satisfied by the analysis condition verification section 304 or a condition related to the method for monitoring performance degradation.

[0101] The correlation degree evaluation result display section 603 displays information related to the evaluation result of the correlation degree evaluation section 501 described later.

[0102] The determination basis display section 604 displays information that is the determination basis of the evaluation result of the correlation degree evaluation section 501. That is, it is possible to display information related to various statistics and the like shown in FIG. 8. Figures 7A-7F

[0103] As described above, by calculating the statistics of the detection frequency of performance degradation for each variation factor and performing screen display, the operator can promote the analysis of the problem at the time of performance degradation.

[0104] Next, the evaluation method using the above-described various statistics will be described. Figure 8 is a flowchart showing an example of the evaluation processing in the correlation degree evaluation section 501, and the description will be made in accordance with this flowchart.

[0105] The correlation degree evaluation section 501 first evaluates whether or not the detection of performance degradation is concentrated when a specific evaluation device or manufacturing device is used (step S801). For example, in the detection frequency of performance degradation evaluated for each predetermined time width for each device ID, in a case where a predetermined number of devices occupying an upper position occupy a predetermined proportion of the total number of detection times of performance degradation, it is determined that it is concentrated when a specific device is used.

[0106] In a case where it is determined that it is concentrated on a specific device (Yes at step S801), it is considered that the correlation degree with the reason inherent to the device is high, and the operator is urged to confirm whether or not there is an abnormality or a problem in the device itself (step S802).

[0107] In a case where it is not determined that it is concentrated on a specific device (No at step S801), the correlation degree evaluation section 501 evaluates whether or not the detection of performance degradation is concentrated on the use conditions of a specific evaluation device or manufacturing device (step S803). For example, in the detection frequency of performance degradation evaluated for each predetermined time width for each device use condition, in a case where a predetermined number of use conditions occupying an upper position occupy a predetermined proportion of the total number of detection times of performance degradation, it is determined that it is concentrated on a specific use condition.

[0108] ​In a case where it is determined that the detection of the performance degradation of the use condition of the specific device is concentrated (step S803: YES), the operator is urged to perform relearning on the basis of the data collected with respect to the use condition (step S804). In addition, it can be confirmed whether the data at the time of learning includes the use condition, and in the case where it does not include the use condition, the operator is urged to perform relearning on the basis of the use condition.

[0109] In a case where it is not determined that the detection of the performance degradation of the use condition of the specific device is concentrated (step S803: NO), the correlation degree evaluation section 501 evaluates whether the detection of the performance degradation is concentrated in a specific product (step S805). For example, the frequency of the detection of the performance degradation evaluated for each predetermined time width is evaluated for each product ID, and in a case where a specific product ID accounts for a predetermined proportion of the total number of detections of the performance degradation, it is evaluated that the detection of the performance degradation is concentrated in the specific product.

[0110] In a case where it is determined that the detection of the performance degradation is concentrated in the specific product (step S805: YES), the operator is prompted about various index values such as the performance degradation detection position, the defect generation position, and the measured value of the product, and the state confirmation of the product is urged (step S806).

[0111] In a case where it is not determined that the detection of the performance degradation is concentrated in the specific product (step S805: NO), the correlation degree evaluation section 501 evaluates whether the detection of the performance degradation is concentrated in a specific position of the product (step S807). This is determined in a case where the detection positions of the performance degradation are concentrated by a predetermined proportion or more in the vicinity of the specific position of the product.

[0112] In a case where it is determined that the detection of the performance degradation is concentrated in the specific position of the product (step S807: YES), the operator is urged to perform relearning because of the lack of learning of the position, or the state of the product is urged to be confirmed (step S808).

[0113] In a case where it is not determined that the detection of the performance degradation is concentrated in the specific position of the product (step S807: NO), the correlation degree evaluation section 501 evaluates whether the detection of the performance degradation is concentrated in a specific model (step S809). For example, the number of detections of the performance degradation is evaluated for each machine learning model ID, and in a case where a predetermined proportion of the number of detections is concentrated in a predetermined number of machine learning models that account for the upper ranks in the frequency of detection, it is determined that there is a concentration with respect to the model.

[0114] In a case where it is determined that the detection of the performance degradation is concentrated in the specific model (step S809: YES), the operator is urged to perform relearning of the specific model (step S810).

[0115] If it is not determined that the detection of performance degradation is concentrated in a specific model (step S809 : No), the operator is urged to re-learn after re-collecting data for all models, or to request a professional engineer to perform problem analysis (step S811 ).

[0116] As described above, by evaluating whether the detection of performance degradation is concentrated on a specific requirement, the factor of performance degradation can be estimated.

[0117] In addition, the relevance evaluation unit 501 Figure 8 In addition to evaluating each candidate sequentially, it is also possible to process all candidate factors in parallel, use the aforementioned statistics to calculate an evaluation value to assess the degree of correlation with performance degradation, and make a decision based on the evaluation value. For example, candidates with a high degree of correlation can be prioritized for confirmation.

[0118] in addition, Figure 8 The various sets of determinations and treatment plan presentations in the process (e.g., steps S801 and S802, steps S802 and S803, etc.) may be performed in a reversed order. For example, after starting (step S800), steps S807 and S808 may be performed first, followed by steps S805 and S806, and so on, thereby reversing the order of the determination processes.

[0119] In addition, the factors of performance degradation can be estimated by methods other than those described here. For example, the factors calculated based on the machine learning model measurement factor 705 and the machine learning model check factor 706 can also be evaluated. Figure 7D When the amount of information shown is large, it is believed that a change has occurred in the machine learning model factors, urging the re-collection of data and the re-learning of the machine learning model 202.

[0120] Furthermore, statistics associated with these may be evaluated, associated as a determination basis, and displayed on the determination basis display unit 604. Attribute information whose statistics differ by a predetermined value or more from those at the time of learning may be identified and presented to the operator or used for relearning.

[0121] In addition, the output unit 502 may also display more than one Figures 7A-7F The statistics used in the analysis shown above can also be changed according to the degree of correlation. For example, the color or font of the screen display can be changed to make the factors with high correlation more conspicuous.

[0122] As described above, according to this embodiment, since the input data 201 is managed in association with various variation factors, it is possible to evaluate statistics for each variation factor when performance degradation of the machine learning model 202 is detected, and the factor of performance degradation can be inferred.

[0123] Example 3

[0124] This embodiment pre-evaluates the suitability (adequacy) of the learning data used when learning the machine learning model 202. The performance of the machine learning model 202 is affected by the distribution of the learning data. Therefore, according to this embodiment, it is possible to pre-estimate the learning data for which the performance of the machine learning model 202 may become insufficient for each piece of attribute information, and to expand the learning data as needed before learning.

[0125] Figure 9 The structure of this embodiment is shown. Figure 9 The parts shown can be similar to those in Example 1 and Example 2 by Figure 1 It can be implemented by application software in the computer 104, or it can be implemented by dedicated hardware. Figure 5 The structure of the second embodiment shown is different in that an adequacy evaluation unit 901 is provided instead of the relevance evaluation unit 501 .

[0126] exist Figure 9 In the structure shown, Figure 3 and Figure 5 The basic functions of the components with the same names and symbols are the same as those in the above embodiment, and detailed description is omitted. In the case of learning data evaluation during learning in this embodiment, the identification information assigning unit 302 assigns whether the data 301 is for specific learning as identification information.

[0127] The analysis condition verification unit 304 determines that the conditions for performing analysis are met when learning is executed. Specifically, at the timing of the next learning start, it is determined that the learning data to be used here has been analyzed in advance. Alternatively, the analysis conditions may be determined to be met if a staff member or the like instructs to perform analysis of the learning data. The attribute information acquisition unit 305, which is classified by identification information, then references the identification information in the storage unit 303 and retrieves from the storage unit 303 the data and attribute information to be used in learning the machine learning model 202 to be trained.

[0128] Then, the statistic calculation unit 306 calculates statistics for each attribute information and identification information acquired by the identification information-classified attribute information acquisition unit 305. The calculation method of various statistics is the same as that described in the previous embodiment and is therefore omitted.

[0129] The sufficiency evaluation unit 901 evaluates the learning data for each predetermined attribute information based on the statistics calculated by the statistics calculation unit 306, and outputs the calculated statistics and the evaluation results to the output unit 502. The operator examines the excess or deficiency of the learning data according to the content displayed by the output unit 502, and performs learning by adding data as needed. Alternatively, after presetting various conditions for the learning data used for learning, additional learning data can be collected to meet these conditions, or additional learning data can be acquired from the storage unit 303, and learning can be automatically executed when these conditions are met.

[0130] Figures 10A-10D The following shows an example of the statistical amount of evaluation and the screen output in the output unit 502 in this embodiment.

[0131] Figure 10A This figure shows an example of a statistic 1001 related to the evaluation device factor. This is an example of evaluating the amount of learning data for each evaluation device or each evaluation condition. In this case, the adequacy evaluation unit 901 evaluates whether a predetermined amount of data exists for each evaluation device or evaluation condition to which the machine learning model 202 is applied. Alternatively, the data amount evaluation may be performed to determine whether there is a bias toward a specific device or condition. This can be achieved, for example, by evaluating the amount of information for a uniform distribution.

[0132] Figure 10B This shows an example of statistics 1002 related to product factors. This is an example of evaluating the distribution of learning data on the product. In this case, the sufficiency evaluation 901 evaluates whether there are more than a predetermined number of learning data at each location on the product. Figure 10B The number of each position is expressed in shades. For example, the positions with less than a predetermined number are darkened, but the positions with more data can also be darkened. In addition, it is also possible to evaluate whether the positions are concentrated on a specific product.

[0133] Figure 10C This figure shows an example of statistics 1003 related to manufacturing equipment factors. This is an example of evaluating the amount of learning data for each manufacturing equipment or each manufacturing condition. In this case, the adequacy evaluation unit 901 evaluates whether a predetermined amount of data exists for each manufacturing equipment or manufacturing condition used in the product to which the machine learning model 202 is applied. Furthermore, when evaluating these data amounts, it is also possible to determine whether there is a bias toward specific equipment or conditions.

[0134] Figure 10DThis figure shows an example of statistics 1004 related to factors that depend on machine learning model 202. This example evaluates a histogram of measured length values ​​when machine learning model 202 is used for product measurement. In this case, the adequacy evaluation unit 901 evaluates whether a predetermined amount of data exists within a predetermined interval of the histogram. Alternatively, a hypothetical histogram can be pre-set based on empirical rules, for example, and the information content of the histogram for the learning data can be evaluated.

[0135] Furthermore, when the machine learning model 202 is used for product inspection, the defect type distribution is evaluated. In this case, the adequacy evaluation unit 901 can evaluate whether a predetermined amount of data exists for each defect type category. Alternatively, a pre-defined distribution of categories can be set based on empirical rules, and the information content of the learning data's category distribution can be evaluated.

[0136] As described above, according to this embodiment, when learning the machine learning model 202, the sufficiency of the learning data can be evaluated in advance from various perspectives. If the sufficiency is evaluated to be insufficient, it can be studied whether additional learning data is needed.

[0137] Example 4

[0138] This embodiment describes performance evaluation after the machine learning model 202 has been learned. According to this embodiment, after the machine learning model 202 has been learned, performance evaluation of each piece of attribute information can be performed, and when applying the machine learning model, application conditions where performance cannot be expected can be determined in advance.

[0139] Furthermore, this allows the operator to be advised of solutions such as expanding the learning data for the condition, not using the machine learning model 202 under the condition, determining whether the machine learning model 202 can be applied, or dividing the application target. Furthermore, it is also possible to suggest further learning or limiting the application target according to predetermined conditions.

[0140] Figure 11 The components shown in FIG10 can be similarly connected to those in Embodiments 1 to 3. Figure 1 It can be implemented by application software in the computer 104, or it can be implemented by dedicated hardware. Figure 9 The difference in the structure of the third embodiment shown is that a performance evaluation unit 1101 for each attribute information is provided instead of the adequacy evaluation unit 901 .

[0141] exist Figure 11 In the structure shown, Figure 3 、 Figure 5 and Figure 9The basic functions of the components with the same names and symbols are the same as those in the previous embodiment, and detailed descriptions are omitted. When performing the performance evaluation after learning in this embodiment, the identification information assigning unit 302 assigns to the data 301 whether it is data used for evaluation of the machine learning model as identification information.

[0142] The analysis condition verification unit 304 determines that the analysis execution conditions are satisfied when the machine learning model 202 completes learning. Alternatively, the analysis conditions may be determined to be satisfied when a staff member or the like instructs to evaluate the machine learning model 202.

[0143] The identification information-based attribute information acquisition unit 305 acquires data used to evaluate the learned machine learning model 202 from the storage unit 303. The statistic calculation unit 306 then calculates a statistic serving as a performance indicator for each piece of attribute information and identification information acquired by the identification information-based attribute information acquisition unit 305.

[0144] The attribute information-based performance evaluation unit 1101 estimates whether the inference performance is insufficient when using the machine learning model 202 based on the statistics calculated by the statistics calculation unit 306. This estimation result is then output to the output unit 502 along with the statistics, prompting the operator to confirm and take appropriate action.

[0145] Figures 12A-12D The statistics calculated in this embodiment and an example of screen output in the output unit 502 are shown.

[0146] Figure 12A In the example, as the evaluation device factor 1201, the performance index of the machine learning model 202 is calculated as a statistic for each evaluation device, and the performance index can also be calculated for each evaluation condition.

[0147] Based on the calculation result, the attribute information-based performance evaluation unit 1101 evaluates whether there are any evaluation devices or evaluation conditions that do not meet the predetermined performance index. If so, the output unit 1102 prompts the operator to take the above-mentioned measures.

[0148] Figure 12B This graph calculates and graphically displays the predicted performance results for each location on the product as a statistic, representing product factors 1202. The attribute-based performance evaluation unit 1101 evaluates whether any location information fails to meet the predetermined performance indicator. If so (in this example, the location with degraded performance enclosed by the dotted line), the output unit 1102 prompts the operator to take the aforementioned measures.

[0149] Figure 12CIn this case, as the manufacturing device factor 1203, the performance index of the machine learning model 202 is calculated as a statistic for each manufacturing device, and the performance index can also be calculated for each manufacturing condition.

[0150] Based on the calculation result, the performance evaluation section 1101 evaluates whether there is no manufacturing device or manufacturing condition that does not achieve the predetermined performance index, distinguished by attribute information, and in the presence thereof, prompts the above-mentioned countermeasures to the operator through the output section 1102.

[0151] Figure 12D is a graph illustrating the measured value (predicted value) at the time of utilization of the machine learning model and the measured value as a target (target value) as the machine learning model factor 1204. At this time, evaluation of the measured value as a target can also be performed to evaluate whether there is a measurement site where the performance of the machine learning model 202 is insufficient, by each measurement site.

[0152] In addition, in the case where the machine learning model is for inspection, the relationship between the inspection prediction result of the machine learning model 202 and the inspection result as a target is evaluated. This can also evaluate, for example, the accuracy, the reproducibility of each defect type, and the precision. In the case where they do not achieve the predetermined performance, the above-mentioned operator is prompted for countermeasures through the output section 1102.

[0153] As described above, according to the present embodiment, it is possible to evaluate, for each attribute information, the model that is estimated to have insufficient performance after learning of the machine learning model 202.

[0154] Furthermore, the present application is not limited to the above-described embodiments, and various modifications are included. For example, the embodiments can also be utilized in combination. In addition, the above-described embodiments are examples that are described in detail in order to facilitate understanding of the present application, and are not necessarily limited to having all the structures described. In addition, a part of the structure of an embodiment can be replaced with the structure of another embodiment, and in addition, the structure of an embodiment can be added with the structure of another embodiment. In addition, with respect to a part of the structure of each embodiment, addition, deletion, and replacement of other structures can be performed.

[0155] Explanation of Reference Signs

[0156] 101…manufacturing device, 102…product, 103…evaluation device, 104…computer.

Claims

1. A data manipulation device that manages and processes utilization data used in a machine learning model that performs predetermined processing on image information of manufactured products, characterized in that: The data operation device comprises: an identification information imparting unit configured to impart identification information to the image information, the identification information being configured to identify whether data processing associated with the machine learning model can be applied; a storage unit that stores the identification information in association with attribute information, the attribute information defining various conditions for acquiring the image information that may become a factor of variation in the utilization data that may affect prediction performance of the machine learning model; an analysis condition verification unit that verifies whether an analysis condition, which is a condition for performing analysis on the machine learning model, is satisfied; an attribute information acquisition unit for each identification information, which acquires the image information and the attribute information associated with the identification information corresponding to the analysis condition from the storage unit when the analysis condition is satisfied; as well as A statistic calculation unit calculates statistics used in the management of the machine learning model based on the acquired image information and attribute information.

2. The data operation device according to claim 1, characterized in that The identification information includes information on whether the machine learning model is used in any of the learning process, evaluation process, and monitoring process, and information on whether the prediction performance is determined to have deteriorated through the monitoring process. The attribute information includes at least one of an acquisition condition, a characteristic condition, and a manufacturing condition. The acquisition condition includes identification information of an evaluation device that acquires the image information or its shooting condition. The characteristic condition is extracted from the image information. The manufacturing condition includes identification information of a manufacturing device that manufactures the manufactured product or its processing condition.

3. The data operation device according to claim 2, characterized in that: The analysis condition verification unit starts analyzing the cause of the degradation when the deterioration of the inference performance of the machine learning model is detected at a frequency greater than or equal to a predetermined frequency. The identification information-based attribute information acquisition unit acquires, from the storage unit, the attribute information associated with the machine learning model in which the deterioration of the inference performance is detected, The statistic calculation unit calculates a predetermined statistic for each piece of attribute information acquired by the identification information-based attribute information acquisition unit. The data operation device includes a relevance evaluation unit that evaluates a relevance of the attribute information and inference performance degradation based on the statistic calculated by the statistic calculation unit.

4. The data operation device according to claim 3, characterized in that: The statistics include any one of the detection frequency of reasoning performance degradation of each evaluation device of the manufactured product, the detection ratio of reasoning performance degradation at each position of the manufactured product, the detection frequency of reasoning performance degradation of each manufacturing device of the manufactured product, the amount of information in the machine learning model as the degree of variation between the data used for inference and the data used for learning, and the detection frequency of reasoning performance degradation of each model of the machine learning model.

5. The data operation device according to claim 4, characterized in that: The data operation device urges the user to determine and collect the data required for re-learning the machine learning model based on the evaluation of the correlation evaluation unit, or urges the user to confirm the status of the evaluation device or the manufactured product or the manufacturing device.

6. The data operation device according to claim 5, characterized in that: The data operation device has an output unit that outputs information related to the machine learning model that detects the degradation of the reasoning performance, the conditions for detecting the degradation of the reasoning performance, the evaluation results in the association evaluation unit, and at least one of the judgment basis in the association evaluation unit including the statistic.

7. The data operation device according to claim 2, characterized in that: The analysis condition verification unit determines that the conditions for performing the analysis are satisfied when the learning of the machine learning model is instructed or when the implementation of the preliminary evaluation related to the adequacy of the learning data is instructed. The identification information-based attribute information acquisition unit acquires, from the storage unit, the learning data and attribute information used in learning the machine learning model. The statistic calculation unit calculates a predetermined statistic based on the learning data and the attribute information thereof acquired by the attribute information acquisition unit classified by identification information. The data operation device includes a sufficiency evaluation unit that evaluates the sufficiency of the learning data of the machine learning model for each piece of attribute information based on the statistic calculated by the statistic calculation unit.

8. The data operation device according to claim 7, characterized in that: The statistics include any one of statistics related to the data volume of the learning data for each of the evaluation devices or each of its evaluation conditions, statistics related to the positional distribution of the learning data on the manufactured product, statistics related to the data volume of the learning data for each of the manufacturing devices or each of its manufacturing conditions, and statistics related to the distribution of the acquired values ​​of the learning data.

9. The data operation device according to claim 8, characterized in that: The adequacy evaluation unit determines and collects data required for learning the machine learning model based on the statistics calculated by the statistics calculation unit to implement learning.

10. The data operation device according to claim 9, characterized in that: The adequacy evaluation unit includes an output unit that outputs at least one of the statistic calculated by the statistic calculation unit and an evaluation result thereof.

11. The data operation device according to claim 2, characterized in that: The analysis condition verification unit determines that the analysis execution condition is satisfied when the learning of the machine learning model is completed or when verification of the machine learning model is instructed. The identification information-based attribute information acquisition unit acquires data and attribute information used in the verification of the machine learning model as an analysis target from the storage unit. The statistic calculation unit calculates a predetermined statistic based on the data and the attribute information thereof acquired by the attribute information acquisition unit for each identification information. The data operation device includes a performance evaluation unit differentiated by attribute information, which evaluates the inference performance of the machine learning model for each attribute information based on the statistic calculated by the statistic calculation unit.

12. The data operation device according to claim 11, characterized in that: The statistics include statistics related to the inference performance of the machine learning model for the evaluation device or each of its evaluation conditions, statistics related to the inference performance of the machine learning model for each position of the manufactured product, statistics related to the inference performance of the machine learning model for the manufacturing device or each of its manufacturing conditions, and at least one of statistics related to the relationship between the predicted value of the machine learning model and the predetermined target value.

13. The data operation device according to claim 12, characterized in that: The performance evaluation unit differentiated by attribute information prompts the user to provide solutions including expansion of learning data for the machine learning model or selection of whether the machine learning model can be applied or an application target based on the evaluation result.

14. The data operation device according to claim 13, characterized in that: The data operation device includes an output unit that outputs at least one of the evaluation result of the performance evaluation unit classified by attribute information and the statistic.

15. A data manipulation method for managing and processing utilization data used in a machine learning model for performing predetermined processing on image information of manufactured products, characterized in that: assigning identification information to the image information, the identification information identifying whether data processing related to the machine learning model can be applied, The identification information is stored in association with attribute information, wherein the attribute information specifies various conditions when acquiring the image information that may become a variable factor of the utilization data that may affect the prediction performance of the machine learning model, verifying whether an analysis condition, which is a condition for performing analysis on the machine learning model, is satisfied, When the analysis condition is satisfied, acquiring the image information and the attribute information associated with the identification information corresponding to the analysis condition; Based on the acquired image information and attribute information, statistics used in the management of the machine learning model are calculated.

Images