Program and information processing apparatus

Abstract

To highly accurately specify and estimate a whole image of factors significantly affecting a final result.SOLUTION: A program causes a computer to perform: a solution enumerating step of enumerating a plurality of approximate solutions of an optimal solution by using a solution random sampling algorithm; and an output step of selecting approximate solutions of the optimal solution exclusively having non-zero coefficient values with respect to respective explanatory variables corresponding to the approximate solutions of the optimal solution and outputting an average value of the respective explanatory variables in a coefficient vector calculated in a set of the approximate solutions of the optimal solution having the non-zero coefficient values, as a final coefficient estimate value.SELECTED DRAWING: Figure 27

Description

[Technical field]

[0001] The present invention relates to a program and an information processing device. [Background technology]

[0002] There is a demand for a mechanism for constructing and utilizing a data-driven model to accurately identify and estimate the "whole picture of factors" that significantly affect the final quality of manufactured goods, purchasing behavior, etc., based on actual data from the industrial and commercial fields. Conventionally, methods for identifying factors have been proposed, such as in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-504772 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, the technical difficulties that make it difficult to identify factors mainly include high dimensions, high correlation, interactions, sequence bias, a mixture of numerical variables and categorical variables, a small number of samples, missing samples, etc. However, in the conventional technology, there is a problem that these technical difficulties cannot be sufficiently solved.

[0005] The present invention has been made to solve the above-mentioned problems, and its purpose is to provide a program and information processing device that can accurately identify and estimate the "overall picture of factors" that significantly affect final results such as the final quality of manufactured products and purchasing behavior, based on actual data from the industrial and commercial fields. [Means for solving the problem]

[0006] One embodiment of the present invention is a program for causing a computer to execute a solution enumeration step of enumerating a plurality of approximation solutions to an optimal solution using a solution random sampling algorithm, and an output step of selecting an approximation solution to an optimal solution that has exclusively nonzero coefficient values ​​for each explanatory variable corresponding to the approximation solution to the optimal solution, and outputting, as a final coefficient estimate, an average value of each of the explanatory variables in a coefficient vector calculated in a set of approximation solutions to the optimal solution that have the nonzero coefficient values.

[0007] Moreover, one embodiment of the present invention is an information processing device including a processor and a storage unit, the processor executing the above-mentioned program. Effect of the Invention

[0008] According to the present invention, it is possible to provide a program and an information processing device that can accurately identify and estimate the "overall picture of factors" that significantly affect the final result of purchasing behavior or the like. [Brief description of the drawings]

[0009] [Figure 1A] FIG. 13 is a diagram showing an example of an image of an optimal solution of L2 norm regularization. [Figure 1B] FIG. 13 is a diagram showing an example of an image of an optimal solution of L1 norm regularization. [Diagram 2] FIG. 1 illustrates an example of a correlation network between variables. [Diagram 3] FIG. 1 is a diagram showing an example of a comparison of the behavior of Lasso, Elastic Net, and CHANOL. [Figure 4] FIG. 1 is a diagram illustrating an example of a representative hierarchical clustering method. [Figure 5A] 1 is a heat map showing correlations for the basic validation dataset 1. [Figure 5B] 1 is a heat map showing correlations for the basic validation dataset 2. [Figure 5C] 1 is a heat map showing correlations for basic validation dataset 3. [Figure 5D]1 is a heat map showing correlations for basic validation dataset 4. [Figure 6A] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the first Lasso relaxivity. [Figure 6B] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the second Lasso relaxivity. [Figure 6C] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the third Lasso relaxivity. [Figure 6D] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the fourth Lasso relaxivity. [Figure 6E] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the fifth Lasso relaxivity. [Figure 6F] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the sixth Lasso relaxivity. [Figure 7A] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the first Lasso relaxivity. [Figure 7B] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the second Lasso relaxivity. [Figure 7C] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the third Lasso relaxivity. [Figure 7D] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the fourth Lasso relaxivity. [Figure 7E] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the fifth Lasso relaxivity. [Figure 7F] FIG. 13 shows an example of a scatter plot of CHANOL solutions at the sixth Lasso relaxivity. [Figure 8A] FIG. 13 is a diagram showing a first example of a scatter plot of Lasso estimates. [Figure 8B] FIG. 13 is a diagram showing a second example of a scatter plot of Lasso estimates. [Figure 8C] FIG. 13 is a diagram showing a third example of a scatter plot of Lasso estimates. [Figure 8D] FIG. 13 is a diagram showing a fourth example of a scatter plot of Lasso estimates. [Figure 9A] FIG. 13 is a diagram showing a first example of a scatter plot of CHANOL estimated values. [Figure 9B] FIG. 13 is a diagram showing a second example of a scatter plot of CHANOL's estimated values. [Figure 9C] FIG. 13 is a diagram showing a third example of a scatter plot of CHANOL's estimated values. [Figure 9D] FIG. 13 is a diagram showing a fourth example of a scatter plot of CHANOL's estimated values. [Figure 10] FIG. 1 is a diagram showing an example of an outline of actual data in a semiconductor manufacturing factory. [Figure 11] FIG. 13 is a diagram showing an example of simulation data in a semiconductor manufacturing factory. [Figure 12] FIG. 13 is a diagram showing an example of variable indexes and partial regression coefficients by correlation level. [Figure 13] FIG. 13 is a diagram illustrating an example of the correspondence between each variable and an evaluation index. [Figure 14] FIG. 13 is a diagram showing the coefficient estimation accuracy of CHANOL when ρ=0.1. [Figure 15] FIG. 13 is a diagram showing the coefficient estimation accuracy of SEHC-Lasso when ρ=0.1. [Figure 16] FIG. 13 is a diagram showing the estimation results of CHANOL when ρ=0.5. [Figure 17] FIG. 13 is a diagram showing the coefficient estimation accuracy of SEHC-Lasso when ρ=0.5. [Figure 18] FIG. 13 is a diagram showing the estimation results of CHANOL when ρ=1.0. [Figure 19] FIG. 13 is a diagram showing the coefficient estimation accuracy of SEHC-Lasso when ρ=1.0. [Figure 20] FIG. 11 is a diagram showing a summary of the number of variables detected by CHANOL and the proposed method (SEHC-Lasso). [Figure 21] FIG. 2 is a diagram showing an example of an output form of SEHC-Lasso according to the present embodiment. [Figure 22A] FIG. 13 is a graph showing the transition of Coef-error, which combines representative variables and correlation variables. [Figure 22B] FIG. [Figure 22C] FIG. 13 is a diagram illustrating the number of detection variables. [Figure 22D] FIG. [Figure 23] FIG. 13 is a diagram showing an example of an estimation result of SEHC-Lasso. [Figure 24] FIG. 13 is a diagram showing an example of an estimation result of ScSEHC-Lasso. [Diagram 25] FIG. 13 is a diagram showing the processing time and number of detected variables for SEHC-Lasso and ScSEHC-Lasso. [Figure 26A] FIG. 13 is a diagram showing the Coef-error combining representative variables and correlation variables. [Figure 26B] FIG. [Figure 26C] FIG. 13 is a diagram showing the number of variable detections. [Figure 26D] FIG. 13 is a diagram showing Recall combining representative variables and correlation variables. [Figure 27] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Figure 28] FIG. 13 is a diagram showing a modified example of the output form of the SEHC-Lasso according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [background] Semiconductor manufacturing is one of the largest manufacturing industries in terms of scale and complexity. In particular, cutting-edge semiconductor manufacturing processes are approaching precision processing of several nanometers and ultra-precise control at the single-atom level, making the manufacturing processes and manufacturing equipment very advanced and expensive. Specifically, a series of processes such as baking, exposure, development, etching, and resist removal are repeated many times, so hundreds to thousands of processes are required to manufacture and ship the final product. Furthermore, each process is composed of multiple devices, and information such as repeated processing, changes in processing units, equipment maintenance, and unscheduled stoppages is also added, so the manufacturing history of each product is diverse. In the semiconductor manufacturing process, it is important to quickly and accurately deal with variations in quality indicators and deterioration of quality, but due to the complexity as described above, there are a huge number of possible causes of incidents, and analysis by quality control experts generally requires a lot of man-hours.

[0012] In semiconductor manufacturing, several hundred pieces of quantitative data, such as temperature, pressure, and gas flow rate, are collected from sensors attached to each piece of semiconductor manufacturing equipment. The total number of variables across all processes is 6 to 10 million, and approximately 2 billion transactions (i.e., about 40 terabytes) are recorded in the database per day. Just as data is utilized in many process industries, the vast amount of recorded data can be actively used for quality control. Utilizing data can contribute to the efficient analysis of defect causes by promoting the integrated use of the specialized knowledge and expertise of quality control experts and data analysis technology.

[0013] In semiconductor manufacturing plants, two types of data are collected: "equipment variables," which are quantitative numerical variables as mentioned above, and "event variables," which are text data that represent the actions taken by on-site engineers (such as changing equipment settings or replacing parts). According to conventional technology, there is a case where 27 factors that affect product quality were successfully extracted from 23,600 equipment variables by using Lasso (least absolute shrinkage and selection operator; lasso or lasso regression; hereinafter simply referred to as Lasso), a type of sparse modeling. However, due to the nature of Lasso, which reduces redundant variables to zero, there is a possibility that variables that should have been retained may have been omitted. In response to this problem, there have been cases in which a system was introduced in which quality control experts reconstruct important variables, reducing the required labor from one week to one day. Specifically, the method involves reconstructing important variables using a correlation network between the variables selected by Lasso and other variables. There remains room for verification and improvement of the effectiveness of CHANOL for high-dimensional data that contains complex correlations and diverse variables, such as those found in real data. In addition, an additional challenge is the problem of selecting ultra-highly correlated variables with correlations greater than 0.9 (especially greater than 0.95), which appear in large numbers in order to efficiently operate mass production factories.

[0014] In this embodiment, the purpose is to develop a variable selection method that takes multicollinearity into account and is applicable to realistic data, and to reduce the oversight of important variables by Lasso and the labor required for variable reorganization.

[0015] First, we explain the regression method for high-dimensional data, as well as the elemental technologies of the proposed method, such as suboptimal Lasso solution enumeration, Group Lasso, and hierarchical clustering. In addition, we outline the algorithms of conventional data analysis cases and explain the problems. Next, we present basic verification to re-examine the effectiveness of previous research and existing algorithms, and use this as material for considering the proposed method. Next, we explain the basic concept of the proposed method based on basic verification, and then explain the contents of the proposed method. Next, we present the results of applying the proposed method to simulation data and discuss their results.

[0016] [About previous research] We will introduce several penalized linear regression methods. For "high-dimensional small sample data" where the number of data samples far exceeds the dimension of the features, such as data obtained in the semiconductor manufacturing process, the estimation of the covariance matrix calculated from the data becomes unstable and the least squares method cannot be applied. In such cases, penalized linear regression using regularization leads to improved prediction performance.

[0017] (1) Ridge regression and Lasso regression Let X∈R^(n×p) and y∈R^n be the explanatory variable matrix and the objective variable vector, respectively. Let the coefficient parameter vector in the linear regression model be β∈R^p (Note that in the text, ^ (hat) represents a power. This applies in the following explanations as well). In this case, the least squares method is formulated as an optimization problem in equation (1). If the estimated amount of the coefficient parameter vector is β^, it can be expressed as in equation (2).

[0018]

number

[0019]

number

[0020] However, as mentioned above, when the data used as explanatory variables is a high-dimensional small sample or when multicollinearity exists between the variables, the covariance matrix (X^TX) becomes irregular and the accuracy of the solution becomes unstable. Optimization with a penalty term is expressed by adding Ω(w), which represents the penalty for the parameter w∈R̂d, to the objective function l(w), as shown in equation (3).

[0021]

number

[0022] Here, λ is a parameter that adjusts the balance between the loss term and the penalty term. In penalized linear regression, similar to equation (3), it is formulated by adding a penalty term to the loss function in equation (1). The penalty term in this case is also called a regularization term, and the concept of "norm", which is a generalization of the length of a vector, is often used. The Lp norm (1≦p≦∞) for vector w is given by:

[0023]

number

[0024] The further w is from the origin, that is, the larger it becomes, so the value becomes larger. In other words, the strong penalty has the effect of contracting w. Penalized linear regression using the L2 norm for the regularization term is called Ridge regression, and that using the L1 norm is called Lasso regression. When the least squares loss is denoted by L(β), the estimators β^ridge and β^lasso of the coefficient parameter vectors of Ridge regression (hereinafter simply referred to as Ridge) and Lasso are defined as optimization problems of equations (5) and (6), respectively.

[0025]

number

[0026]

number

[0027] Here, λ (≧0) is a regularization parameter that adjusts the strength of regularization. By adding the L2 norm regularization term to the least squares loss, the Ridge estimator becomes ((X^T X+λI)^(-1) X^T y). (X^T X+λI) is always regular and has an inverse matrix. In other words, a stable estimator can be obtained even if the data is a high-dimensional small sample. In addition, the absolute value of the predicted value of the parameter can be suppressed. As a result, the variance of the predicted value is reduced, leading to improved prediction accuracy. In addition, suppressing overfitting may lead to improved generalization performance for unknown data. When the L1 norm regularization term is used, in addition to suppressing the predicted value of the parameter, the number of non-zero parameters can be suppressed. In other words, the number of variables included in the model is reduced, and improved interpretability can be expected. A major difference between the two methods is the parameter estimation for correlated explanatory variables.

[0028] FIG. 1A is a diagram showing an example of an image of an optimal solution of L2 norm regularization. FIG. 1B is a diagram showing an example of an image of an optimal solution of L1 norm regularization. Figures 1A and 1B show the parameter estimation of Ridge and Lasso when two correlated variables are used as explanatory variables. The equivalence set of the loss function is represented by an elliptical contour, and the optimal solutions are represented by points P1 and P2.

[0029] When the L2 norm is used for regularization, the feasible region of β becomes a circle centered on the origin (Figure 1A), while when the L1 norm is used, it becomes a diamond centered on the origin and with its vertex on the axis (Figure 1B). As a result, in Lasso, the optimal solution is likely to exist on the axis, so one of the values ​​is estimated to be 0, that is, variable selection is performed. In addition, due to its geometric properties, variables that are not necessary for prediction can be estimated to be 0 regardless of whether they are correlated, making it possible to build a model with high interpretability. In contrast, in Ridge, the optimal solution is not necessarily likely to exist on the axis, so active variable selection does not occur. Lasso has the advantages mentioned above, but this means that only one of the variables is selected when there are highly correlated variables or variables in a multicollinear relationship. Therefore, it has been pointed out that there is a possibility that truly important variables may be overlooked.

[0030] (2) Elastic Net As a selection method to avoid overlooking important variables, there is Elastic Net, which uses both L1 norm and L2 norm for the regularization term. The estimator β^EN of the coefficient parameter vector of Elastic Net is defined as the optimization problem of Equation (7).

[0031]

number

[0032] Here, λ (≧0) is a regularization parameter, and α (0≦α≦1) is a parameter that adjusts the ratio between the L1 norm regularization term and the L2 norm regularization term. The Elastic Net corresponds to the Lasso when α = 1, and to the Ridge when α = 0. The L1 norm strengthens the sparseness of the solution, while the L2 norm promotes the group effect for correlated variables. Specifically, it assigns coefficients with similar values ​​to correlated variables. It also ensures that the same coefficient value is assigned to variables with the same value. The Elastic Net avoids active variable selection for correlated variables due to the group effect, while underestimating and selecting regression coefficients in order to share the true effect on the target variable between correlated variables. As a result of this underestimation, the importance of variables may be misidentified.

[0033] (Initiatives at semiconductor manufacturing plants) Figure 2 shows an example of a correlation network between variables. Using Lasso, we have succeeded in extracting 27 factors that affect product quality from 23,600 equipment variables obtained during the manufacturing process. However, since data obtained during the semiconductor manufacturing process contains many variables with strong correlations, Lasso's nature of reducing redundant variables to zero means that variables that should be important may not be selected due to the correlation between variables. Therefore, after automatic selection by Lasso, we investigate similar variables by creating and referencing a correlation network between variables (Figure 2). Although the above process significantly reduced the time required to search for defect factors, it still requires more than one day of labor.

[0034] (Lasso relaxation solution enumeration) The application of CHANOL (Convex Hull Approximation of Nearly Optimal Lasso), a method for enumerating suboptimal Lasso solutions, to semiconductor manufacturing data has been proposed in order to reduce the amount of work required and avoid overlooking important variables. CHANOL is an extended Lasso model proposed to overcome the Lasso's issue of overlooking important variables and to improve the validity and interpretability of the model. By enumerating a small number of diverse suboptimal Lasso solutions (Lasso relaxation solutions), it is possible to select a wide range of features that are in a multicollinear relationship, making it easier for humans to examine the validity of the model. Details of the CHANOL algorithm are described in Algorithm 1.

[0035] (Algorithm 1)Convex Hull Approximation of Nearly Optimal Lasso <Step1:Random sampling step> We sample M suboptimal Lasso solutions from B(ν) by solving equation (10) using a binary tree search. Let S be the set of solutions obtained.

[0036]

number

[0037]

number

[0038]

number

[0039] <Step2:Selection step> By greedily solving equation (11), we enumerate K solutions such that Q ⊆ S.

[0040]

number

[0041] Here, the Hausdorff distance is defined by equation (12).

[0042]

number

[0043] In the above formula, M is the number of Lasso relaxation solutions sampled, K is the number of Lasso relaxation solutions selected, α is the Lasso relaxation degree (α>1), and ρ is the regularization parameter (ρ>0). X, y, β, β*, and ν* are defined by the following formula.

[0044]

number

[0045] In CHANOL, in addition to sampling a set S that sufficiently approximates the convex set of Lasso relaxation solutions, it calculates a small number of solution sets Q that approximate S by solving the problem of minimizing the Hausdorff distance, which is an index of distance between sets. Since the solution set obtained by minimizing the Hausdorff distance has elements that are far apart, it is possible to obtain a diverse set of solutions that sufficiently approximate the Lasso relaxation solution set.

[0046] It has been pointed out that CHANOL has the feature of listing various solutions, and highly correlated variables are exclusively incorporated into the solution (estimation model), making it possible to select variables with strong correlations without overlooking them and to perform equal-level estimation. The "equal-level estimation" described in this embodiment refers to a state in which the effect on the objective variable is not shared between correlated variables and the effect is correctly estimated by a single variable. The performance of Lasso, Elastic Net, and CHANOL is compared using Relative Impact (hereinafter also referred to as RI) (Equation (13)), which evaluates the relative impact of each variable excluding contraction due to regularization, and Mean Relative Impact Error (hereinafter, MRIE) (Equation (14)), which is the average error from the ideal value. As a result, as shown in Figure 3, it was found that CHANOL can correctly estimate the relative impact of each important variable compared to other methods.

[0047]

number

[0048]

number

[0049] Figure 3 shows an example of a comparison of the behavior of Lasso, Elastic Net, and CHANOL. Previously, it has been shown that CHANOL has more stable coefficient estimation accuracy than Lasso and Elastic Net, guidelines for setting CHANOL hyperparameters have been presented, and the applicability of CHANOL to problems of estimating and selecting correlated explanatory variables at the same level has been demonstrated. However, the subjects of the numerical experiments were limited to simple artificial data sets. The data used in semiconductor manufacturing plants contains both quantitative and qualitative variables, and the correlations between explanatory variables are complex, with some having correlations exceeding 0.95. However, previous studies did not take these conditions into account. Therefore, the applicability to real data or simulated data is still unclear, and there is room for verification. In this embodiment, the purpose is to develop a variable selection method that takes multicollinearity into account and is applicable to realistic data, and to reduce the oversight of important variables by Lasso and the labor required for variable reorganization. The solution is to estimate the same level including correlated variables, rather than just the important variables alone. To achieve this, we first apply CHANOL to the complex conditions that appear in real data, and re-examine its effectiveness and issues. Next, we develop and verify a new method to overcome the issues.

[0050] (Group Lasso) Group Lasso is a type of variable selection method, and unlike Lasso, it promotes sparsity for a given variable group. Elastic Net also has a group effect, but the grouping of explanatory variables in Elastic Net is based on correlation, and it has been pointed out that the effect appears automatically during the calculation process. In contrast, Group Lasso assumes that the group structure of explanatory variables is known. Therefore, information about the group structure must be included in the model. Group Lasso is formulated as shown in equation (15). The regularization term used in the second term of this equation is generally called the L1 / L2 regularization term.

[0051]

number

[0052] βg is a partial vector obtained by limiting β to elements contained in group g∈G, and cg>0 represents the weight of group g. The larger the value of cg, the more likely it is that variables contained in that group are simultaneously reduced to 0. In the L1 / L2 regularization term, a penalty based on the L2 norm is applied to explanatory variables contained in the same group, and a penalty based on the L1 norm is applied to relationships between groups. In other words, while sparsity is promoted on a group-by-group basis, when a group is selected, the coefficients of all explanatory variables contained in that group are more likely to be nonzero.

[0053] (Hierarchical Clustering) Clustering is a general term for a method of defining the distance between analysis targets and classifying the targets according to the closeness of the distance. Clustering targets include various things such as samples and explanatory variables, but in the fields of variable selection and feature engineering, it is often used to group explanatory variables. After clustering of explanatory variables, variables belonging to the same group are replaced with one representative variable, and new features are synthesized using average values ​​and principal components, thereby reducing redundant explanatory variables. In the present embodiment, clustering is also used to group explanatory variables.

[0054] Clustering methods can be broadly divided into hierarchical and non-hierarchical methods. Non-hierarchical methods include the k-means method, which determines the cluster to which a cluster belongs based on the distance from a randomly determined cluster center. However, non-hierarchical clustering is generally a method in which the number of clusters is determined in advance and objects are classified into the determined number of clusters, so it is highly likely that it is not suitable for operation data in semiconductor manufacturing factories where the number of clusters is unknown. In fact, when the k-means method is applied in the basic verification of this embodiment, the number of clusters changes depending on the initial value, and clustering is performed based on that, so that variables that do not belong to each correlation group are forcibly put into a group, making it impossible to accurately estimate the correlation group. Therefore, in this embodiment, we have decided to focus on the hierarchical clustering method.

[0055] Figure 4 shows an example of a typical hierarchical clustering method. As shown in the figure, there are four typical hierarchical clustering methods with different distance index calculation mechanisms. The algorithm for hierarchical clustering follows the procedure shown in Algorithm 2.

[0056] (Algorithm 2)Hierarchical Clustering <step1> Determine an index to measure the distance between individual objects. Calculate all the distances between individual objects, and integrate the objects with the smallest distance into the first cluster.

[0057] <step2> Calculate the distance between all newly formed clusters and the objects, and merge the one with the smallest distance between the objects. Repeat this process until all clusters are merged.

[0058] <step3> A dendrogram showing the cluster integration process is drawn, and the objects are divided into multiple groups by separating the clusters at any distance.

[0059] (Basic verification) We verify the behavior of CHANOL using multiple simulation data sets that capture the characteristics of real data. First, we explain the purpose and direction of the basic verification. Next, we provide an overview of the simulation data sets created for the basic verification, and then show the results of each verification. Finally, we summarize the basic verification and present the effectiveness and challenges of CHANOL.

[0060] (Objectives and directions of basic verification) As described above, in this embodiment, we aim to estimate the important variables at the same level, not just the important variables alone, but also the variables correlated with them. The purpose of the basic verification of this embodiment is to clarify whether the original CHANOL can achieve the same level of estimation of correlated variables even when there is diversity in the data, which was not considered in the past. To this end, data reflecting various variable settings, such as correlations that appear in real data, effects on the objective variable, and a mixture of qualitative and quantitative variables, are artificially created to verify the behavior of CHANOL. In basic verifications 1 to 3, the hyperparameters M and K, which adjust the number of relaxed solution samples and the number of selections of CHANOL, are fixed to 500 and 50, respectively. In addition, in the basic verifications, the hyperparameter ρ, which adjusts the strength of regularization, is set to 0.3. These were set to levels that were considered appropriate based on prior verification.

[0061] (Summary of basic verification simulation data) The following describes the simulation data used in basic verifications 1 to 3. Figures 5A to 5D are heat maps showing the correlation between explanatory variables (x_n in the figure, n=[1, 10]) and objective variables (target in the figure). FIG. 5A is a heat map showing correlations for the base validation dataset 1. FIG. 5B is a heat map showing correlations for the base validation dataset 2. FIG. 5C is a heat map showing correlations for the base validation dataset 3. FIG. 5D is a heat map showing correlations for the base validation dataset 4.

[0062] The type of hatching indicates the standard of correlation, and on the heat map, a faint variable relationship indicates a strong correlation, and a dark variable relationship indicates a weak correlation. In all datasets, the number of samples is n=1000, and the dimension of the explanatory variable is p=100. In basic validation datasets 1 to 3, all variables are generated from a multivariate normal distribution, and in basic validation dataset 4, only variables that have an effect on the objective variable and variables that are correlated with them are generated from a binomial distribution Binomial(n,{0.95,0.50,0.05}). Hereafter, variables that truly have an effect on the objective variable in the statistical model are referred to as "representative variables," and variables that are correlated with the representative variables are referred to as "correlated variables."

[0063] In the basic validation dataset 1 shown in Figure 5A, a model is constructed with multiple representative variables that have different degrees of effect on the objective variable. The variables {x_0, x_2, x_4, x_6} are assigned regression coefficients of {2.0, 4.0, 6.0, 8.0}, respectively, and the variables {x_1, x_3, x_5, x_7} are assigned as correlated variables with a correlation of 0.99.

[0064] The basic validation dataset 2 shown in Figure 5B is composed of multiple representative variables with the same effect on the objective variable and correlation variables with different degrees of correlation with them. The regression coefficient of 3.0 is given equally to the variables {x_0, x_2, x_4, x_6, x_8}, and the correlations of the variable pairs {(x_0, x_1), (x_2, x_3), (x_4, x_5), (x_6, x_7), (x_8, x_9)} are set to {0.99, 0.95, 0.90, 0.85, 0.80}, respectively.

[0065] The basic validation datasets 3 and 4 shown in Figures 5C and 5D are composed of multiple representative variables with similar effects on the objective variable. A regression coefficient of 3.0 is given to the representative variables {x_0, x_2, x_4}, and the variables {x_1, x_3, x_5} are given as correlation variables with a correlation of -1 to -0.99. The difference between datasets 3 and 4 is whether the explanatory variables of interest are numerical variables representing quantitative variables or 01 variables representing qualitative variables (categorical variables). In dataset 3, {x_0, x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9} are numerical variables, and in dataset 4, {x_0, x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9} are 01 variables.

[0066] (Basic Verification 1) In basic verification 1, we verify the effect of differences in the influence of each representative variable on the objective variable on CHANOL estimation. The CHANOL estimation values ​​for Dataset 1 shown in Figure 5A are shown in the scatter plots of Figures 6A to 6F. FIG. 6A shows an example of a scatter plot of CHANOL solutions at the first Lasso relaxivity. FIG. 6B is an example of a scatter plot of the CHANOL solution at the second Lasso relaxivity. FIG. 6C shows an example of a scatter plot of the CHANOL solution at the third Lasso relaxivity. FIG. 6D shows an example of a scatter plot of the CHANOL solution at the fourth Lasso relaxivity. FIG. 6E shows an example of a scatter plot of the CHANOL solution at the fifth Lasso relaxivity. FIG. 6F shows an example of a scatter plot of the CHANOL solution at the sixth Lasso relaxivity. In each of Figures 6A to 6F, the vertical axis represents the representative variable, and the horizontal axis represents the coefficient estimate of the correlation variable. The Lasso relaxation degree, which is a CHANOL parameter, is increased from Figure 6A to Figure 6F.

[0067] In Figures 6A to 6F, the CHANOL coefficient estimates are plotted, with the representative variable on the vertical axis and the correlated variable estimate on the horizontal axis. Additionally, the Lasso relaxation degree α, one of the hyperparameters, is increased from Figure 6A to Figure 6F. From the results in Figures 6A to 6F, it can be seen that the relaxation degree at which the coefficient estimates of the representative variable and its correlated variable can be stably obtained at the same level differs depending on the coefficient value of the representative variable. In the setting of α = 1.01 shown in Figure 6C, x_0 and x_1, i.e., highly correlated variables, are exclusively included in the solution (estimated model). Hereafter, an estimation in which such highly correlated variables are exclusively included in the solution (i.e., one of them is estimated to be zero) is called an "exclusive estimation." Next, under the setting of α=1.1 shown in FIG. 6D, exclusive estimation is performed for the variable pairs x_0 and x_1, as well as x_2 and x_3. When exclusive estimation values ​​are obtained, it can be seen that the representative variables and the correlation variables are evaluated at the same level. In this embodiment, the goal is to obtain exclusive estimation values ​​for all representative variables and correlation variables. However, exclusive estimation values ​​are obtained for the variable pairs x_4 and x_5, and x_6 and x_7, which have relatively large regression coefficient values. In the case of α=2.5 (FIG. 6F), a problem occurs in which many solutions other than the exclusive solutions are obtained for the variable pairs x_0 and x_1, and x_2 and x_3, which have relatively small regression coefficient values. In such solutions, large regression coefficient values ​​with different signs are obtained between correlated variables, and the true effect is estimated while canceling out the effects. For example, if the true value of the regression coefficient for a representative variable is 2.0, a non-exclusive solution would give estimates such as 20 and -18 for the representative and correlated variables, respectively, and the sum of these would give the true value of 2.0. Hereafter, such solutions will be referred to as "sharing solutions." As a result of such sharing solutions being included in the enumeration solution, there is a possibility that the true importance of the variables will be misjudged. As stated above, the results of Basic Verification 1 show that the degree of relaxation required to obtain an exclusive solution that does not include a sharing solution and is composed of coefficient estimates at the same level varies depending on the correct coefficient value.

[0068] (Basic Verification 2) In basic verification 2, we verify the impact of the difference in correlation between the representative variable and the correlated variable on the CHANOL estimation. The CHANOL estimation values ​​using Data Set 2 shown in Figure 5B are shown in the scatter plots of Figures 7A to 7F. FIG. 7A shows an example of a scatter plot of CHANOL solutions at the first Lasso relaxivity. FIG. 7B is an example of a scatter plot of the CHANOL solution at the second Lasso relaxivity. FIG. 7C shows an example of a scatter plot of the CHANOL solution at the third Lasso relaxivity. FIG. 7D shows an example of a scatter plot of the CHANOL solution at the fourth Lasso relaxivity. FIG. 7E shows an example of a scatter plot of the CHANOL solution at the fifth Lasso relaxivity. FIG. 7F shows an example of a scatter plot of the CHANOL solution at the sixth Lasso relaxivity. 7A to 7F, the vertical axis represents the representative variable, and the horizontal axis represents the coefficient estimate of the correlation variable. The Lasso relaxation degree, which is a CHANOL parameter, is increased from FIG. 7A to FIG. 7F.

[0069] In Figures 7A to 7F, the CHANOL coefficient estimates are plotted as in Figures 6A to 6F, and the Lasso relaxivity is increased from Figure 7A to 7F. The results in Figures 7A to 7F show that the relaxivity at which the coefficient estimates of the representative variable and its correlated variable are stably obtained at the same level varies depending on the degree of correlation between the representative variable and the correlated variable. In Fig. 7C, when α = 1.01, exclusive estimation is performed for x_0 and x_1, i.e., for variable pairs with relatively strong correlations. In addition, when α ≥ 1.1, exclusive solutions are obtained for all pairs of representative variables and correlated variables, but at the same time, many shared solutions are also selected (Figs. 7D to 7F). Since the sharing solution when α=1.1 is an element contained inside the relaxed solution set, it is thought possible to prevent its selection by reducing the number of selected relaxed solutions, K. On the other hand, since the sharing solution when α≧1.5 is an element contained on the convex hull boundary surface of the relaxed solution set, it is impossible to prevent its selection even by reducing K. Thus, the results of Basic Verification 2 show that the degree of relaxation at which exclusive solutions can be obtained differs depending on the degree of correlation between the representative variables and the correlated variables.

[0070] (Basic Verification 3) In Basic Verification 3, we verify the effect of the variable form on CHANOL estimation. To this end, we use Dataset 3 shown in Figure 5C and Dataset 4 shown in Figure 5D. Figures 8A to 8D and Figures 9A to 9D show scatter plots of Lasso and CHANOL estimates, respectively. The stars in the figures indicate the Lasso solution, the white circles indicate the CHANOL sampling solution (the result of STEP 1), and the black circles indicate the CHANOL selection solution (the result of STEP 2). Also, Figures 8A, 8B, 9A, and 9B show the results when α = 1.05, Figures 8C, 8D, 9C, and 9D show the results when α = 1.50, Figures 8A, 8C, 9A, and 9C show the results when ρ = 0.01, and Figures 8B, 8D, 9B, and 9D show the results when ρ = 0.50. The coefficient values ​​of the representative variables and the magnitude of correlation of the correlated variables are set equal in Data Set 3 and Data Set 4. Meanwhile, regarding the representative variables and correlated variables, Data Set 3 is composed of quantitative variables, and Data Set 4 is composed of qualitative variables. From Figures 8A to 8D and Figures 9A to 9D, it can be seen that the degree of relaxation at which the coefficient estimates of the representative variables and their correlated variables can be stably obtained at the same level differs depending on the form of the explanatory variables. When Dataset 3 is used, the most stable and exclusive solution is obtained in Figure 8B, that is, when α = 1.05, ρ = 0.50. On the other hand, when Dataset 4 is used, the exclusive solution is obtained in Figure 9C, that is, when α = 1.50, ρ = 0.01. This shows that when the important variables to be estimated are a mixture of quantitative and qualitative variables, it is difficult to obtain stable and exclusive estimates for all representative and correlated variables using a model with a single parameter set.

[0071] (Summary of basic verification) From the basic verification, it was found that when using CHANOL, the optimal hyperparameter setting for exclusive estimation differs depending on the regression coefficient, the level of correlation, and the numerical characteristics of the explanatory variables. This indicates that in real data with various conditions, CHANOL estimation using a single parameter set is unstable and there is a high possibility that an estimated value that deviates from the correct answer value will be obtained, and this point is positioned as a problem of CHANOL in this embodiment. On the other hand, the effectiveness of CHANOL was also confirmed from the basic verification. In many cases, as shown in Figures 6F and 9C, an exclusive solution is selected at the same time as the sharing solution. In other words, if the sharing solution can be eliminated, it is possible to obtain a stable estimated value with a model using a single parameter set even in a situation where various data conditions are mixed. Below, a method for overcoming the problems of CHANOL revealed from the basic verification described above will be proposed and explained in detail.

[0072] (Method of this embodiment) Based on the results of the basic verification, we thought that if we could focus on each correlation group from the CHANOL sampling solutions and select only exclusive solutions within the group, we could achieve the solution policy for the problem of this embodiment, which is simultaneous selection and equal-level estimation of representative variables and correlated variables. In this chapter, we propose a method based on this hypothesis. The algorithm of the proposed method is shown in Algorithm 3. Here, the proposed method is named SEHC-Lasso (Simultaneous estimation of highly correlated variables - Lasso).

[0073] (Algorithm 3)SEHC-Lasso (Simultaneous estimation of highly correlated variables - Lasso)

[0074] <step1>Random sampling (CHANOL Step1) By solving the above equation (10) using a binary tree search, M suboptimal Lasso solutions are sampled.

[0075] <step2>Feature selection Select xi∈R^N that satisfies equation (16). This extracts representative variables and correlation variables. Let Q be the index set of the extracted variables.

[0076]

number

[0077] <step3>Feature clustering Variable clusters are constructed by hierarchical clustering using the maximum distance method, with the inv-correlation defined in equation (17) as the index. Equation (18) is the merging condition. Let Gk (Gk ⊂ Q) be the index set of variables belonging to each cluster.

[0078]

number

[0079]

number

[0080] <step4>Coefficient evaluation The left side of equation (19) is the final estimate.

[0081]

number

[0082] In the above formula, M is the number of Lasso relaxed solution samples, α is the Lasso relaxation degree (α>1), T1 is the threshold for variable selection, T2 is the threshold for merging clusters, and ρ is the regularization parameter (ρ>0). In addition, N is the number of samples, β^iq is the coefficient estimate for the solution q of variable xi, and ρab is the correlation between variables (a, b).

[0083] That is, in the method of this embodiment, first, in Step 1, a random solution sampling algorithm is used to enumerate multiple Lasso relaxation solutions, similar to Step 1 of CHANOL. Step 1 is affected by M, which is an adjustment parameter for the number of relaxation solution samples, but here, a sufficiently large number is specified so that exclusive estimates can be obtained for representative and correlation variables under any conditions. In this embodiment, the Lasso relaxation solution has been described as an example of the approximation solution of the optimal solution, but is not limited thereto. As the approximation solution of the optimal solution, any of the Elastic Net relaxation solution, the Ridge regression relaxation solution, or any other relaxation solution of a regression method using an L1 norm regularization term or an L2 norm regularization term, or any relaxation solution of a multiple regression analysis may be used. In other words, these exemplified solutions may be listed as the approximation solution of the optimal solution in the above-mentioned Step 1.

[0084] In Step 2, representative and correlated variables are extracted based on the enumerated relaxed solutions. Specifically, for each variable, variable selection is performed by setting a threshold for the absolute average of non-zero estimates. The average here refers to the average for the enumerated solution set for each variable. Focusing on the average for the enumerated solutions, it was observed that representative and correlated variables that have an effect on the objective variable, as well as variables with strong correlation pairs, tend to have a larger absolute average as the degree of relaxation α increases. As mentioned in the basic verification, sharing solutions are sampled between strongly correlated variables. In addition, the coefficient value of the sharing solution increases with the opposite sign as the degree of relaxation increases. As a result, the absolute average also increases. For a single representative variable that does not have a correlated variable, the sampling area of ​​the solution expands as the degree of relaxation increases, making it easier to obtain excessive estimates. On the other hand, the coefficient estimates of variables that do not have an effect on the objective variable and do not have a strong correlation concentrate around zero, so the absolute average does not increase even if the degree of relaxation is increased. Therefore, it was thought that by setting a threshold for the absolute average of non-zero estimates, it would be possible to extract representative and correlated variables. Due to the nature of obtaining a set of variables whose average absolute value of the relaxed solution is above the threshold, the relaxation level α needs to be set large for Step 2. However, this is consistent with the entire process because the relaxation level α needs to be set to a large value so that exclusive estimates can be obtained for representative and correlated variables under any conditions.

[0085] In Step 3, clustering is performed using the inverse of the correlation between variables as an index, and correlation groups of the variables are estimated. Information on the estimated correlation groups is used in Step 4 to search for exclusive solutions. In this embodiment, when a certain variable in a group is focused on, the state in which that variable is the only one estimated to be non-zero is called an exclusive estimation. If the correlation groups are understood, it is easy to select only the exclusive estimations therein. Here, the reason why hierarchical clustering is used is that, as described in Section 2.5, the problem setting is one in which the number of clusters is unknown. In addition, the longest distance method is used as the distance metric because it is possible to configure clusters with the strength of correlation arbitrarily specified by the hyperparameter T2.

[0086] In Step 4, exclusive solutions are selected for each variable within each correlation group, and the average value calculated for the set of exclusive solutions is used as the final estimate.

[0087] The whole method of this embodiment is as shown in Algorithm 3 above. However, as shown in the numerical verification results described later, the proposed method of Algorithm 3 has a processing time problem originating from the original CHANOL. In particular, when there are many strong correlations between variables and the dimensions of the data are large, the random sampling step of Step 1 requires a lot of calculation time. In order to solve this problem and improve the accuracy and stability, we propose variable screening shown in Algorithm 4 as a preprocessing of the proposed method.

[0088] (Algorithm 4) Screening <step1>Feature clustering (SEHC-Lasso Step3) Variable clusters are constructed by hierarchical clustering using the maximum distance method, with the inv-correlation defined in equation (17) as the index. The merging condition is equation (18). The index set of variables belonging to each cluster is Gk (Gk ⊂ Q).

[0089] <step2>Group Lasso Group Lasso is performed using group information Gk.

[0090] <step3>Feature selection Select xi ∈ R^N such that β^i ≠ 0. The selected xi forms a dataset to be input to the subsequent algorithm.

[0091] where T2 is the cluster merging threshold, ρ is the regularization parameter (ρ>0), N is the number of samples, and β^iq is the coefficient estimate of the variable xi in the solution q.

[0092] This preprocessing is aimed at reducing the input variables to the subsequent SEHC-Lasso, mainly excluding incorrect variables in advance. This data dimensionality reduction is expected to reduce the processing time required for relaxed solution sampling. Group Lasso was adopted instead of Lasso as the variable selection method. In this embodiment, we aim to achieve the same level of estimation and selection of representative variables and correlated variables. Therefore, not only representative variables must be left at the screening stage, but also correlated variables must be selected at the same time. In Group Lasso, variables belonging to the same group are simultaneously estimated to be non-zero or zero. Due to this characteristic, we thought that by using the correlation group structure of variables as group information for Group Lasso, it would be possible to efficiently extract a group of variables that affect the target variable.

[0093] In addition, in Group Lasso, coefficients are estimated by sharing the effect between variables belonging to the same group. Therefore, the estimation results of preprocessing do not result in the same level estimates for representative variables and correlated variables, but this is not a problem because this preprocessing is used only for prior variable selection and the estimated values ​​are not used.

[0094] (Numerical verification) In this chapter, we use simulation data that mimics real data to verify the behavior of the proposed method and its superiority over previous research. First, we explain the purpose and direction of the numerical verification. Next, we provide an overview of the dataset created for the verification and show the results of each verification. Finally, we summarize the verification results and show the achievements of the proposed method.

[0095] (Objectives and directions of numerical verification) The numerical verification in this chapter aims to apply the proposed method to a data set containing a variety of conditions that mimics real data, and to verify the selection and estimation accuracy of representative variables and correlated variables. In numerical verification 1, we compare CHANOL to verify the accuracy of variable selection and coefficient estimation. In numerical verification 2, we confirm the behavior of the proposed method with respect to hyperparameters such as α and ρ. In Numerical Verification 3, we introduce preprocessing to address the issue of processing time described below, and present the results of verifying the accuracy of variable selection and coefficient estimation for the entire series of analysis processes, including preprocessing, as well as the processing time.

[0096] (Summary of numerical verification simulation data) The simulation data used in the numerical verifications 1 to 3 will be explained. FIG. 10 is a diagram showing an example of an outline of actual data in a semiconductor manufacturing factory. Fig. 11 is a diagram showing an example of simulation data from a semiconductor manufacturing factory. The simulation data was created based on interviews with semiconductor manufacturing factories. First, the actual data from semiconductor manufacturing plants includes both quantitative and qualitative variables. Equipment variables, which are quantitative variables, refer to numerical data obtained by sensing voltage, pressure, temperature, etc., but standardization is essential because there are various units and numerical ranges. For this reason, simulation data is generated based on a multivariate standard normal distribution. On the other hand, categorical variables, which are qualitative variables, indicate the details of procedures performed by engineers, processes, equipment, etc. All categorical variables are one-hot encoded and expressed as two values, 0 and 1. For this reason, simulation data is also generated from a binomial distribution as binary data of 0 / 1 for each variable.

[0097] In the simulation data set, the number of samples is n=1000, the dimension of the explanatory variables is p=5000, and it is composed of the same number of instrument variables and categorical variables. In other words, there are 2500 instrument variables and 2500 categorical variables. Of the 5000 variables, 24 variables are set as representative variables, and each representative variable forms a correlation group with the correlation variables according to the correlation shown in Table 5-2. In addition to the representative variables and correlation variables, non-detection target variables that have no effect on the objective variable are also generated and integrated as noise variables. The strength of correlation between the representative variables and the correlation variables is set to four levels: no correlation, low correlation, medium correlation, and high correlation. In addition, the partial regression coefficients that represent the effect on the objective variable are set to {2.5, 5.0, 10.0}, and are given as the effect of the representative variables. Each of the four correlation groups is configured to include three representative variables with different degrees of influence. The correlation groups belonging to no correlation, low correlation, and medium correlation are composed of five variables, including one representative variable and four correlation variables. In addition, the correlation group belonging to high correlation consists of 10 variables, including one representative variable and nine correlation variables. The objective variable y∈R^n is generated by equation (20).

[0098]

number

[0099] The first to third terms in equation (20) represent the representative variables of the instrument variables, and the fourth to sixth terms represent the representative variables of the categorical variables. The seventh term, ε, is an error term, which is generated from a normal distribution so that the SN ratio calculated by equation (21) becomes a specified value (here, 5.0).

[0100]

number

[0101] Based on the above, the overall variable indexes and partial regression coefficients by correlation level are summarized in Figure 12 . FIG. 12 is a diagram showing an example of variable indexes and partial regression coefficients by correlation level.

[0102] (Numerical verification 1) In numerical verification 1, we compare CHANOL to verify the accuracy of variable selection and coefficient estimation of the proposed method. As mentioned in Chapter 4, in the proposed method, a large value must be set for the relaxation degree α so that exclusive estimates can be obtained for representative and correlated variables under any conditions. Therefore, in numerical verification 1, we run each algorithm with α = 1.5. We also set M, a hyperparameter for adjusting the number of relaxed solution samples, to 400, and the hyperparameters related to variable selection and clustering to (T1 = 1.0, T2 = 0.4). For fairness, we also run the variable selection step using T1 for CHANOL, which is the comparison target.

[0103] (Evaluation method) In the numerical verification 1, the algorithm is evaluated based on the indices of formulas (22) to (26). TP (True Positive) and FN (False Negative) respectively represent the number of correctly detected variables among the detection targets and the number of undetected variables. FP (False Positive) and TN (True Negative) respectively represent the number of misdetected variables among the non-detection targets and the number of undetected variables. The recall and precision in formulas (22) and (23), which are calculated using these indices, respectively represent the proportion of correctly detected variables among the detection targets and the proportion of the detection targets among the detected variables. The coef-error defined in formula (25) indicates the average error rate of the estimated value in a certain variable set, and the closer the value is to zero, the more accurate the coefficient estimation is. In formula (25), V(v∈V) is the variable set, (coefv*) is the correct value of the regression coefficient, and (coefv^) is the estimated value of the regression coefficient. The noise-error defined in equation (26) is the average of the coefficient estimates of the undetected variables, and its ideal value is zero.

[0104]

number

[0105]

number

[0106]

number

[0107]

number

[0108]

number

[0109] In numerical verification 1, the algorithm is evaluated according to the evaluation policies shown in (a) to (d) below. a) The above evaluation indicators are calculated from simulation data generated using five different random number seeds, and evaluated using the average value. b) In anticipation of use in semiconductor manufacturing plants, in addition to representative variables, correlation variables belonging to a group of highly correlated variables are targeted for detection and estimation. Other correlation variables are treated as noise variables. c) For estimates of unestimated variables in the proposed method (described later), select non-zero estimates from the Step 1 sampling solution and complement their average values. d) Calculate the evaluation index according to the circles in Figure 13. FIG. 13 is a diagram showing an example of the correspondence between each variable and the evaluation index.

[0110] (Verification results) 14 to 19 show the evaluation results of each technique. Figures 14 to 15 show the coefficient estimation accuracy of CHANOL and the proposed method (SEHC-Lasso) when ρ = 0.1. Comparing the two tables, the parts that show higher accuracy are written in bold. Figures 14 to 15 show that, as intended, the proposed method has improved the estimation error of the representative variables and correlation variables that are the detection targets compared to CHANOL. Looking at the correlation variables, the error rate Coef-Error has improved significantly from 0.777 to 0.419. In addition, Noise-Error, which indicates the average estimated value of the noise variables, is also closer to zero.

[0111] Figures 16 and 17 show the estimation results of CHANOL and the proposed method (SEHC-Lasso) when ρ = 0.5. As with the case of ρ = 0.1, the proposed method achieves relatively high accuracy in many evaluation indicators. Next, when comparing the proposed methods, when ρ = 0.1, the Coef-Error is larger for representative variables and smaller for correlated variables. This is thought to be the result of the strong penalty leading to underestimation of representative variables, and the increased exclusivity of estimation for correlated variables.

[0112] Figures 18 and 19 show the estimation results of CHANOL and the proposed method (SEHC-Lasso) when ρ = 1.0. As mentioned above, the proposed method achieved relatively high accuracy in many evaluation indices. Next, when comparing the proposed methods with each other, the Coef-Error, which combines the representative variables and correlation variables, is smallest among the three levels of ρ when ρ = 1.0.

[0113] Figure 20 summarizes the number of variables detected by CHANOL and the proposed method (SEHC-Lasso) when α = 1.5, ρ = {0.1, 0.5, 1.0}. Both methods detected a number of variables far exceeding the number of target variables, 78. This leads to the low precision in Figures 14 to 19. With the proposed method, it was confirmed that the number of false positives tends to increase as ρ increases.

[0114] The results of Numerical Test 1 show that the proposed method achieves estimation accuracy that exceeds that of CHANOL for both representative and correlated variables. In particular, the improvement in accuracy of correlated variable estimation is remarkable. In addition, for estimating noise variables that are not to be detected, the method outputs values ​​closer to zero than CHANOL. This shows that even when noise variables are detected, it is easier to distinguish them from important variables such as representative and correlated variables.

[0115] Here, we show the results of the verification of the estimation accuracy for all variables. Next, we focus on individual variables from the estimation result output of the proposed method and confirm that simultaneous estimation of representative variables and correlated variables has been achieved.

[0116] (Output of the method of this embodiment) First, we explain the output format of the proposed method, SEHC-Lasso. Fig. 21 is a diagram showing an example of the output form of SEHC-Lasso in this embodiment. Fig. 21 shows an excerpt of an output table of the assumed estimation results. In this figure, pred_value is the detection variable index, pred_group is the estimation group, and pred_coef is the regression coefficient estimate. The density of the frame of pred_value is changed according to the correct regression coefficient value so that it can be visually grasped.

[0117] In the output of the method SEHC-Lasso of this embodiment, an estimated group index is also output so that the variable group can be easily understood. From FIG. 21, it can be seen that only representative variables are correctly detected and estimated for variable indexes 0 to 40 belonging to the medium correlation or lower, and representative variables as well as correlation variables are correctly detected and estimated for variable indexes 45 to 74 belonging to the high correlation variable group. In addition, the goal of estimating representative variables and correlation variables at the same level is also achieved. On the other hand, the existence of some variables for which estimated values ​​are not output is cited as an issue. For example, in the group of the high correlation variable group with the correct regression coefficient value of (5,10), estimates of some variables are not obtained. This is thought to be due to the strength of correlation. In the high correlation variable group, the correlation with respect to the representative variable is set to {0.999, 0.975, 0.95, 0.925, 0.9, 0.875, 0.85, 0.825, 0.8} in ascending order of variable index. From this, it is believed that exclusive solutions were not sampled for variables with relatively weak correlation, and therefore no estimates were obtained. On the other hand, in the group with a correct regression coefficient value of 2.5, estimates were obtained for all variables. This difference is because, in the sampling algorithm of Step 1, the larger the correct regression coefficient value, the more likely it is that a representative variable will be selected. Due to this tendency, in the group with a relatively large correct regression coefficient value (5,10), the representative variable has the highest probability of being included in the enumerated solution, followed by the correlated variable according to the strength of the correlation. On the other hand, in the group with a relatively small correct regression coefficient value, the difference in the probability that each variable will be included in the enumerated solution becomes smaller. As a result, it is believed that exclusive estimates were obtained for all variables. In Numerical Verification 1, as described in the evaluation strategy, for variables for which estimates cannot be obtained, non-zero estimates are selected from the Step 1 sampling solution, and their average values ​​are complemented to obtain evaluation values.

[0118] (Numerical verification 2) In numerical verification 2, we consider the optimal settings of hyperparameters (α, ρ) for the proposed method from three perspectives: regression coefficient estimation accuracy, variable selection accuracy, and processing time. The values ​​shown in numerical verification 2 are the average values ​​of evaluation values ​​calculated from simulation data generated from five different random number seeds, as in numerical verification 1. The characteristics of the estimation results for (α, ρ) are shown in Figures 22A to 22D.

[0119] Figure 22A shows the trend of Coef-error, which is the sum of the representative variables and the correlation variables. From this result, we can see that the larger ρ is, the smaller the estimation error becomes, and there is a tendency for it to decrease monotonically. 22B shows the processing time, which can be seen to be shorter as ρ increases and α decreases. FIG. 22C shows the number of detected variables, and it can be seen that the smaller ρ and α are, the smaller the number of detected variables tends to be. In FIG. 22A, it can be seen that the Coef-error tends to be small when α=1.1, but this is thought to be due to overlooking variables. When the relaxation degree α is small, as described in Section 5.3.3, detection is performed but no estimate is obtained, and variables with relatively weak correlations within the correlation group tend not to be detected. Therefore, there is no need to complement the estimate, and the Coef-error of the representative variable and the correlation variable combined is small. From the results in FIG. 22A to FIG. 22C, it is considered that the setting value of α is appropriate at 1.1, but as shown in FIG. 22D, when α=1.1, the Recall is low. Specifically, even at the best, the Recall is 0.9 or less, which indicates that about 10% of important variable candidates are overlooked. In this situation, important variables may be overlooked regardless of multicollinearity, so this does not meet the problem and purpose of this embodiment. On the other hand, looking at Figure 22C, at α=2.0, the number of false positives far exceeds the number of variables to be detected, and interpretability is significantly reduced. Based on the above results, from the perspective of variable selection, the appropriate α under the conditions set in this experiment is considered to be 1.5, because it keeps the number of detections low and also reduces the oversight of important variable candidates. In addition, looking at Figure 22A, the difference in estimation error between α=1.5 and α=2.0 is not large, except when ρ is 0.1. Therefore, a setting of α=1.5 is considered appropriate from the perspective of coefficient estimation accuracy as well. Next, let us consider ρ. First, looking at Figs. 22A to 22B, when ρ=0.1, the coefficient estimation accuracy is low and the processing time is long. Therefore, it can be said that ρ=0.1 is not suitable. Also, looking at Fig. 22C, when α=1.5, the number of variables detected when ρ is 1.0 is more than double that when it is 0.5. In other words, from the viewpoint of variable selection and model interpretability, it can be said that a setting value of ρ of 0.5 is more suitable than 1.0. From the above discussion, it is considered that the hyperparameter settings for the proposed method should be in the neighborhood of (α, ρ) = (1.5, 0.5).

[0120] (Numerical verification 3) In Numerical Test 3, we take up variable screening as a measure to address the two issues of processing time and low precision mentioned in the previous section, and verify the performance of a series of analyses that integrate screening and SEHC-Lasso. First, we compare the performance of SEHC-Lasso and SEHC-Lasso with screening (hereinafter referred to as ScSEHC-Lasso) under the optimal hyperparameter setting (α, ρ) = (1.5, 0.5) in the previous section. Next, we re-examine the behavior of ScSEHC-Lasso for (α, ρ). Figures 23 and 24 show the estimation results of SEHC-Lasso and ScSEHC-Lasso. Note that Figure 23 shows the same results as Figure 17. Figure 24 also compares the number of detected variables and processing time. Comparing Figures 23 and 24, we can see that the ScSEHC-Lasso with added variable screening has improved indicators for variable selection. This is thought to be due to the reduction in incorrect variables by variable screening.

[0121] The results also confirmed that variable screening using correlation clustering and Group Lasso properly identified the representative and correlated variables to be detected, and reduced unnecessary variables. Figure 25 shows the processing time and number of detected variables for SEHC-Lasso and ScSEHC-Lasso. This shows that ScSEHC-Lasso significantly reduces processing time compared to SEHC-Lasso. In addition, the reduction in the number of detections (a 42.6% reduction) has also improved the interpretability of the model. Next, the optimal setting of hyperparameters (α, ρ) for ScSEHC-Lasso will be considered. The characteristics of the estimation results of ScSEHC-Lasso for (α, ρ) are shown in Figures 26A to 26D. As in Numerical Verification 2, Figure 26A shows the Coef-error combining representative variables and correlation variables, Figure 26B shows the processing time, Figure 26C shows the number of variables detected, and Figure 26D shows the Recall combining representative variables and correlation variables. As in SEHC-Lasso, the three indices Coef-error, processing time, and number of variables detected show good results when α=1.1, while the Recall is relatively low. Therefore, when the detection of important variable candidates is emphasized, it is considered appropriate to set α to 1.5. When α is set to 1.5, it is considered appropriate to select the setting value of ρ to 0.5, as in SEHC-Lasso. ρ=0.5 is the inflection point of the monotonic function (increase / decrease) in the graphs of Figures 26A to 26C, where the improvement in the indicators decreases for Coef-error and processing time, and the number of detected variables is just before a significant increase in the number of detected variables. On the other hand, if some oversight is acceptable, it may be possible to set α to 1.1. In that case, the most suitable ρ is 1.0, which shows the best results for all indicators.

[0122] (Summary of numerical verification) Numerical verification 1 showed that the proposed method, SEHC-Lasso, achieved better accuracy than CHANOL in regression coefficient estimation and variable selection. It was also shown that not only did it surpass the prior art in accuracy, but it also achieved the same level of estimation of representative variables and correlated variables in a data set with a mixture of various conditions such as regression coefficients, correlations, and numerical properties. In numerical verification 2, we verified the behavior of the proposed method with respect to the hyperparameters (α, ρ) and concluded that the optimal setting was near (α, ρ) = (1.5, 0.5). On the other hand, issues were raised regarding the processing time and number of variable detections in numerical verification 1 and numerical verification 2. In response to this, ScSEHC-Lasso, which adds variable screening by Group-Lasso, significantly reduced the processing time and succeeded in suppressing the number of variable detections without lowering Recall. From the above, it has been demonstrated that ScSEHC-Lasso can achieve short-time calculation of simultaneous selection and same-level estimation of representative variables and correlation variables, which is the solution policy for the problem of this embodiment, and that it is worth proceeding to verification with real data.

[0123] In this embodiment, the objective is to develop a variable selection method that takes multicollinearity into account and is capable of handling realistic data scale and complexity, and to reduce the oversight of important variables by Lasso and the labor required for variable reorganization. In order to achieve these two goals simultaneously, we considered a strategy of selecting and estimating at the same level not only the important variables alone, but also the correlation variables that are correlated with the important variables. We proposed a new method (SEHC-Lasso) that combines hierarchical clustering using the longest distance method with CHANOL, a method for enumerating suboptimal Lasso solutions, with the aim of simultaneously selecting and estimating important and correlated variables to the same level. In addition, to address issues of false detection of variables and processing time pointed out in the proposed method, we devised a variable screening method that combines Group Lasso, a type of linear regression with structured penalties, with correlation clustering. SEHC-Lasso first enumerates multiple Lasso relaxation solutions using a solution random sampling algorithm, similar to Step 1 of CHANOL. Next, important variable candidates are detected by setting a threshold for the absolute average of non-zero estimates for each variable. Then, clustering is performed using the inverse of the correlation between the detected variables as the distance index to estimate correlation groups of the variables. After that, exclusive solutions are selected for each variable within each correlation group, and the average calculated in the set of exclusive solutions is used as the final estimate. In variable screening, first, correlation groups of variables are estimated by correlation clustering used in Step 3 of SEHC-Lasso. Then, by using the correlation group structure of variables as group information for Group Lasso, variable groups that affect the target variable are extracted in advance. This aims to reduce calculation time. To verify the method, we used simulation data created based on interviews at semiconductor manufacturing plants. The verification using the simulation data showed that the proposed method, SEHC-Lasso, achieved better accuracy than CHANOL in regression coefficient estimation and variable selection. In addition to exceeding the accuracy of prior art, the technology was also shown to achieve the same level of estimation for representative and correlated variables in a data set containing a mixture of diverse conditions, including numerical and categorical variables, regression coefficients, correlations, and numerical characteristics. In addition, ScSEHC-Lasso, which added variable screening, significantly reduced processing time and succeeded in suppressing the number of detected variables without lowering Recall. As a result, SEHC-Lasso and ScSEHC-Lasso achieved the solution to the problem of this embodiment, which is simultaneous selection and equal-level estimation of representative variables and correlation variables.

[0124] (Device configuration) 27 is a diagram showing an example of a configuration of the information processing system 1 of this embodiment. The information processing system 1 includes an information processing device 10 and an output device 20. The information processing device 10 is a so-called computer device, and includes a processor 110 and a storage unit 120. The storage unit 120 stores a program for executing each of the above-mentioned steps, and various data acquired from an external device (e.g., a semiconductor manufacturing device). The processor 110 obtains the above-mentioned various calculation results by processing the various data stored in the storage unit 120 based on the program stored in the storage unit 120. Note that the illustration and description of input operation devices such as a keyboard, mouse, and touch panel are omitted in the figure. The output device 20 includes, for example, a liquid crystal display and a printer, and outputs the results of processing by the information processing device 10.

[0125] A specific description will now be given of the processing performed by the processor 110. The processor 110 executes a solution enumeration step, a candidate detection step, a correlation group estimation step, and an output step.

[0126] The solution enumeration step corresponds to Step 1 of the above-mentioned Algorithm 3. The solution enumeration step is a step for enumerating multiple approximate solutions of the optimal solution using a solution random sampling algorithm. Here, the solution random sampling algorithm is, for example, an algorithm corresponding to Step 1 of the above-mentioned CHANOL.

[0127] The approximation solution to the optimal solution is, for example, a Lasso relaxation solution. As described above, the explanatory variables of the input data set for the solution random sampling algorithm are p-dimensional, and the number of objective variables is n.

[0128] As shown in the above numerical verification, the explanatory variables may include quantitative variables or binarized variables, and the objective variables may be characteristic values ​​related to quality or condition.

[0129] That is, in the solution enumeration step, processor 110 executes an algorithm equivalent to Step 1 of CHANOL to enumerate multiple Lasso relaxation solutions.

[0130] The candidate detection step corresponds to Step 2 of the above-mentioned Algorithm 3. The candidate detection step is a step of detecting candidates of important explanatory variables (e.g., representative variables and correlation variables) by setting a predetermined threshold for the average absolute value of non-zero coefficient estimates for each explanatory variable corresponding to an approximate solution (e.g., Lasso relaxation solution) of the optimal solution enumerated in the solution enumeration step. Here, the predetermined threshold is, for example, the above-mentioned T1. That is, in the candidate detection step, the processor 110 selects xi ∈ RN that satisfies equation (16) and sets a predetermined threshold T1 to extract representative variables and correlation variables.

[0131] The correlation group estimation step corresponds to Step 3 of the above-mentioned Algorithm 3. The correlation group estimation step is a step in which clustering is performed using as an index the inverse of the correlation (Equation (17)) between each explanatory variable corresponding to an approximate solution (e.g., Lasso relaxation solution) of the optimal solution enumerated in the solution enumeration step, and a correlation group of each explanatory variable is estimated. That is, in the correlation group estimation step, the processor 110 executes clustering using the formula (17) as an index.

[0132] The output step corresponds to Step 4 of the above-mentioned Algorithm 3. The output step is a step of selecting an approximation of the optimal solution having a nonzero coefficient value exclusively for each explanatory variable corresponding to an approximation of the optimal solution (e.g., Lasso relaxation solution), and outputting the average value (Equation (19)) of each explanatory variable in the coefficient vector calculated in the set of approximation of the optimal solution having the nonzero coefficient value as a final coefficient estimate to, for example, the output device 20. That is, in the output step, processor 110 outputs the value shown in equation (19) as the final coefficient estimate.

[0133] Here, the processor 110 outputs the coefficient estimates to the output device 20 in the output format shown in Fig. 21. As a result, the coefficient estimates are output from the output device 20 in the output format shown in Fig. 21. As described above, in Fig. 21, pred_value is the index of the extracted variable (i.e., the detected variable index), pred_group is the estimated group (i.e., the estimated group), and pred_coef is the regression coefficient estimate. The density of the frame of pred_value is changed according to the correct regression coefficient value so that it can be visually understood.

[0134] That is, the information processing device 10 outputs (e.g., to the output device 20) as an output dataset any one of an identifier for the set of important explanatory variable candidates (e.g., a detection variable index), a cluster to which the important explanatory variable candidates belong (e.g., an estimation group) which is the estimation result of the correlation group, coefficient estimates of explanatory variables other than the important explanatory variable candidates, or coefficient estimates of the important explanatory variable candidates.

[0135] Furthermore, the information processing device 10 may output the screening result (or the screening condition obtained in the screening process) described in Algorithm 4 (variable screening) to the output device 20. That is, the information processing device 10 may output at least one of the correlation group of explanatory variables estimated by correlation clustering and the group of explanatory variables extracted from the execution result of Group Lasso using the structure of the correlation group.

[0136] [Variation 1] The processor 110 may additionally execute the following steps S101 to S103. (Step S101) An arbitrary number of explanatory variables, including zero, are selected from the correlation group. (Step S102) An estimation model is derived from the selected explanatory variables. (Step S103) A response variable is estimated using the estimation model and the explanatory variables.

[0137] 28 is a diagram showing a modified example of the output form of SEHC-Lasso in this embodiment. This figure shows an example of the results selected in step S101 described above. In this example, it is shown that No.={0 to 8, 10, 26, 29} has been selected. Note that the selection may be automatic (i.e., selection is performed by processor 110 based on a program) or manual selection using an input operation device (not shown). In the case of automatic selection, the processor 110 selects based on coefficient values ​​or correlations (for example, the variable with the largest coefficient value, the variable with the highest correlation with the objective variable, a combination of these, etc.). Note that there may be a variable group in which no variables are selected.

[0138] The processor 110 uses equation (27) to derive the estimated value Y^ (n×1 dimension) of the response variable.

[0139]

number

[0140] In the derivation of the estimated value, σ = 0 is generally used. In other words, the noise variable ε is not added. Here, considering a special practical case, equation (28) is shown in a form including ε.

[0141] [Variation 2] Also, the processor 110 may additionally execute the following step S201. (Step S201) The residual between the estimated value of the dependent variable and the theoretical value or correct value of the dependent variable is calculated. Here, the residual (residual vector r) is given by equation (28).

[0142]

number

[0143] In other words, the information processing device 10 outputs an estimated value of the objective variable to the output device 20 by using the estimation model or the Group Lasso estimation model and the explanatory variable data.

[0144] [Variation 3] Also, the processor 110 may additionally execute the following step S301. (Step S301) An estimation model and explanatory variables generated based on a probability distribution or random numbers are used as input to generate an estimate of a new objective variable.

[0145] An example of generating new estimates of the response variable (i.e., new samples) will be described. The new samples include one or more pairs of (Yk~, Xk~) generated by equation (29).

[0146]

number

[0147] When the signal-to-noise (S / N) ratio is specified, σ is determined so that the S / N ratio satisfies equation (30).

[0148]

number

[0149] The newly generated data are pairs (Y~, X~) shown in equation (31), which are obtained by integrating samples for all selections k.

[0150]

number

[0151] Note that equation (29) includes an error term ε. That is, the processor 110 may execute a step of estimating a response variable using an estimation model (or an estimation model of the Group Lasso), a predetermined probability distribution, and explanatory variables given by random numbers, and adding an error term ε to the response variable.

[0152] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. The configurations described in the above-mentioned embodiments may be combined.

[0153] Each unit of each device in the above-described embodiments may be realized by dedicated hardware, or may be realized by a memory and a microprocessor.

[0154] In addition, each part of each device may be composed of a memory and a CPU (central processing unit), and the functions of each part of each device may be realized by loading a program for realizing the function of each part into the memory and executing the program.

[0155] In addition, a program for implementing the functions of each unit of each device may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing by each unit of the control unit. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.

[0156] In addition, if a WWW system is used, the "computer system" also includes the homepage providing environment (or display environment). In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes those that dynamically hold a program for a short period of time, such as communication lines when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0157] 1...information processing system, 10...information processing device, 20...output device, 110...processor, 120...storage unit

Claims

1. On the computer, A solution enumeration step of enumerating a plurality of approximation solutions to the optimal solution using a solution random sampling algorithm; an output step of selecting an approximation of an optimal solution having a non-zero coefficient value exclusively for each explanatory variable corresponding to the approximation of the optimal solution, and outputting an average value of each explanatory variable in a coefficient vector calculated in a set of approximation of the optimal solution having the non-zero coefficient value as a final coefficient estimate; A program for executing the above.

2. a candidate detection step for detecting important explanatory variable candidates by setting a predetermined threshold value for the average absolute value of non-zero coefficient estimates for each of the explanatory variables corresponding to the approximate solutions of the optimal solution enumerated in the solution enumeration step; The program according to claim 1 , further comprising:

3. a correlation group estimation step of performing clustering using the inverse of the correlation between each of the explanatory variables corresponding to the approximate solutions of the optimal solution enumerated in the solution enumeration step as an index, and estimating a correlation group of each of the explanatory variables; The program according to claim 2, further comprising:

4. Prior to the solution enumeration step, a correlation group estimation step of estimating correlation groups of the explanatory variables by correlation clustering; a Group Lasso execution step of executing Group Lasso by using the correlation group structure of the explanatory variables as group information for Group Lasso; A screening step of extracting a group of explanatory variables that affect the target variable from the execution result of the Group Lasso in advance; The program according to claim 1 , further comprising:

5. The approximation of the optimal solution is one of the relaxation solutions of Lasso relaxation solution, Elastic Net relaxation solution, Ridge regression relaxation solution, other regression methods using L1 norm regularization term or L2 norm regularization term, or multiple regression analysis. The program according to any one of claims 1 to 3.

6. The input data set for the solution random sampling algorithm has p-dimensional explanatory variables and n-dimensional objective variables. The program according to any one of claims 1 to 3.

7. The explanatory variables include quantitative variables or binarized variables. The program according to claim 6.

8. The objective variables are characteristic values ​​related to quality and condition. The program according to claim 6.

9. The output data set is one of the set of candidates for the important explanatory variables, the correlation group to which the candidates for the important explanatory variables belong, the coefficient estimates of explanatory variables other than the candidates for the important explanatory variables, or the coefficient estimates of the candidates for the important explanatory variables. The program according to claim 3.

10. selecting any number of explanatory variables, including zero, from the correlation set; deriving an estimation model from the selected explanatory variables; estimating a response variable using the estimation model and the explanatory variables; The program according to claim 3, further comprising:

11. A step of calculating a residual between the estimated value of the dependent variable and a theoretical value or a correct value of the dependent variable. The program according to claim 10, further comprising:

12. Using the estimation model and explanatory variables generated based on a probability distribution or random numbers as input, an estimate of a new objective variable is generated. The program according to claim 10.

13. At least one of a correlation group of the explanatory variables estimated by the correlation clustering and a group of explanatory variables extracted from a result of executing Group Lasso using the structure of the correlation group is generated. The program according to claim 4.

14. An estimated value of a target variable is output using an estimation model derived from the explanatory variables or an estimation model of the Group Lasso and the explanatory variables. The program according to claim 4.

15. A step of estimating a response variable using an estimation model derived from the explanatory variables or the estimation model of the Group Lasso and explanatory variables given by a predetermined probability distribution or random numbers, and adding an error term to the response variable. The program according to claim 4, further comprising:

16. An information processing device including a processor and a storage unit, An information processing device, wherein the processor executes the program according to claim 1 .

Images