A model information determination method, device and system

Abstract

Embodiments of the present application provide a model information determination method, device and system. The method comprises: obtaining each hyperparameter of a target business model to be trained, and data element features and evaluation indexes used for training the target business model; obtaining a causal relationship between each hyperparameter of the target business model and the evaluation indexes; and iteratively adjusting each hyperparameter of the target business model according to the data element features, the evaluation indexes and the causal relationship, to obtain an adjusted target hyperparameter set. Embodiments of the present application determine a hyperparameter having a positive influence on evaluation indexes through a causal relationship between the hyperparameters and the evaluation indexes, improve the precision of model parameter optimization, avoid a large number of calculation experiments, thereby reducing the cost of model parameter optimization and reducing the occupation of system resources in the business model training phase.

Description

Technical Field

[0001] This invention relates to the field of big data analysis, and in particular to a method, apparatus and system for determining model information. Background Technology

[0002] With the increasing informatization and intelligence of society, using search space to train business models and then using the trained business models to achieve intelligent processing of big data business has gradually become a common practice in the big data industry.

[0003] In current model training processes, the corresponding hyperparameters are typically searched from the search space for training. Existing search methods include model-free search methods, evolutionary algorithm-based search methods, and Bayesian optimization-based search methods. These existing hyperparameter search methods require extensive computational experimentation, resulting in huge system resource consumption, high model parameter optimization costs, and low accuracy. Summary of the Invention

[0004] This invention provides a method, apparatus, and system for determining model information, which solves the problems of existing hyperparameter search methods requiring extensive computational experiments, resulting in huge system resource consumption, high model parameter optimization costs, and low accuracy.

[0005] To address the aforementioned technical problems, the present invention provides a method for determining model information, the method comprising:

[0006] Obtain the hyperparameters of the target business model to be trained, as well as the data meta-features and evaluation metrics used to train the target business model;

[0007] Obtain the causal relationship between each hyperparameter of the target business model and the evaluation index;

[0008] Based on the data element features, the evaluation indicators, and the causal relationships, the hyperparameters of the target business model are iteratively adjusted to obtain the adjusted target hyperparameter set.

[0009] Optionally, in the above method, the step of iteratively adjusting each hyperparameter of the target business model based on the data element features, the evaluation indicators, and the causal relationships to obtain the adjusted target hyperparameter set includes:

[0010] Based on the causal relationship, at least one candidate hyperparameter to be adjusted is determined;

[0011] Based on the evaluation index, obtain the causal effect evaluation model that corresponds one-to-one with each of the candidate hyperparameters;

[0012] The data element features and the candidate hyperparameters are processed using the respective causal effect evaluation models to obtain the intervention effect corresponding to each candidate hyperparameter and the evaluation index.

[0013] Based on the effects of each intervention, the candidate hyperparameters are iteratively adjusted.

[0014] Optionally, in the above method, the step of processing the data meta-features and the candidate hyperparameters using the respective causal effect evaluation models to obtain the intervention effect corresponding to each candidate hyperparameter and the evaluation index includes:

[0015] For each causal effect assessment model, the candidate hyperparameters corresponding to the causal effect assessment model are used as intervention variables, the data element features and other hyperparameters besides the candidate hyperparameters used as intervention variables are used as adjustment variables, and the assessment index is used as the target variable.

[0016] The intervention variable and the adjustment variable are input into the causal effect evaluation model corresponding to the intervention variable to obtain the first intervention effect corresponding to the intervention variable.

[0017] Optionally, in the above method, the step of iteratively adjusting each hyperparameter of the target business model based on the data element features, the evaluation indicators, and the causal relationships to obtain the adjusted target hyperparameter set includes:

[0018] Based on each of the first intervention effects, determine the adjustment method for the intervention variable corresponding to each of the first intervention effects;

[0019] Based on the adjustment method corresponding to each intervention variable, the intervention variables of each causal effect assessment model are adjusted;

[0020] Each of the aforementioned causal effect assessment models was reassessed based on the adjusted intervention variables to obtain an optimized set of hyperparameters;

[0021] The target business model is trained and estimated based on the optimized hyperparameter set and the data element features;

[0022] If the model convergence or termination condition is met, the model training and estimation are terminated, and the optimized hyperparameter set is determined as the target hyperparameter set.

[0023] Optionally, in the above method, the step of re-evaluating the causal effect based on the adjusted intervention variables to obtain an optimized hyperparameter set includes:

[0024] Each of the aforementioned causal effect assessment models is reassessed based on the adjusted intervention variables to obtain the second intervention effect corresponding to the adjusted intervention variables;

[0025] Based on the second intervention effect, the intervention variable that has the greatest intervention effect on the evaluation index is obtained, and the optimal intervention variable is obtained.

[0026] Based on the optimal intervention variable corresponding to each of the aforementioned causal effect assessment models, an optimized set of hyperparameters is obtained.

[0027] Optionally, in the above method, the convergence or termination condition includes at least one of the following: reaching the set number of iterations and satisfying the target value set by the target evaluation index.

[0028] Optionally, the method further includes:

[0029] Construct the causal effect assessment model.

[0030] Optionally, in the above method, the step of constructing the causal effect assessment model includes:

[0031] Obtain the original training datasets and corresponding historical model training data for different scenarios; wherein the historical model training data includes at least hyperparameters and evaluation metrics;

[0032] Extract data metadata features from the original training dataset;

[0033] A modeling parameter tuning result table is generated based on the data element features and the historical model training data.

[0034] Based on the modeling and parameter tuning results table and the causal algorithm, the model is trained to generate a causal effect evaluation model.

[0035] Optionally, in the above method, the step of training the model to generate a causal effect evaluation model based on the modeling parameter tuning result table and the causal algorithm includes:

[0036] One hyperparameter is selected from the modeling and parameter tuning result table as an intervention variable, and one evaluation index is selected as the target result. The unselected hyperparameters and data element features in the modeling and parameter tuning result table are used as the set of adjustment variables.

[0037] The causal algorithm is trained based on the intervention variables, the target outcome, and the set of adjustment variables to obtain the causal evaluation model.

[0038] Optionally, in the above method, the step of extracting data meta-features from the original training dataset includes:

[0039] Random sampling is performed on the original training dataset to obtain multiple data subsets;

[0040] The meta-feature extractor is used to extract features from the multiple data subsets to obtain multiple meta-feature vectors;

[0041] The average value of the multiple meta-feature vectors is calculated to obtain the data meta-features.

[0042] This invention also provides a model information determination device, the device comprising:

[0043] The first acquisition module is used to acquire the various hyperparameters of the target business model to be trained, as well as the data element features and evaluation metrics used to train the target business model.

[0044] The second acquisition module is used to obtain the causal relationship between the various hyperparameters of the target business model and the evaluation index;

[0045] The iterative adjustment module is used to iteratively adjust each hyperparameter of the target business model based on the data element features, the evaluation indicators, and the causal relationships, to obtain the adjusted target hyperparameter set.

[0046] Optionally, in the above apparatus, the iterative adjustment module includes:

[0047] The first determining submodule is used to determine at least one candidate hyperparameter to be adjusted based on the causal relationship.

[0048] The first acquisition submodule is used to acquire, based on the evaluation index, a causal effect evaluation model corresponding one-to-one with each of the candidate hyperparameters.

[0049] The first causal effect assessment submodule is used to process the data element features and the candidate hyperparameters using the respective causal effect assessment models to obtain the intervention effect corresponding to each candidate hyperparameter and the assessment index.

[0050] The iterative adjustment submodule iteratively adjusts each of the candidate hyperparameters based on the effects of each intervention.

[0051] Optionally, in the above apparatus, the first causal effect evaluation submodule is specifically used for:

[0052] For each causal effect assessment model, the candidate hyperparameters corresponding to the causal effect assessment model are used as intervention variables, the data element features and other hyperparameters besides the candidate hyperparameters used as intervention variables are used as adjustment variables, and the assessment index is used as the target variable.

[0053] The intervention variable and the adjustment variable are input into the causal effect evaluation model corresponding to the intervention variable to obtain the first intervention effect corresponding to the intervention variable.

[0054] Optionally, in the above-described apparatus, the iterative adjustment module further includes:

[0055] The second determining submodule is used to determine the adjustment method of the intervention variable corresponding to each of the first intervention effects based on each of the first intervention effects;

[0056] The adjustment submodule is used to adjust the intervention variables of each of the causal effect assessment models based on the adjustment methods corresponding to each intervention variable.

[0057] The second causal effect assessment submodule is used to re-evaluate the causal effect of each of the causal effect assessment models based on the adjusted intervention variables, so as to obtain an optimized hyperparameter set;

[0058] The model training and estimation submodule is used to train and estimate the target business model based on the optimized hyperparameter set and the data element features;

[0059] The third determining submodule is used to terminate model training and estimation if the model convergence or termination conditions are met, and to determine the optimized hyperparameter set as the target hyperparameter set.

[0060] Optionally, in the above apparatus, the second causal effect evaluation submodule is specifically used for:

[0061] Each of the aforementioned causal effect assessment models is reassessed based on the adjusted intervention variables to obtain the second intervention effect corresponding to the adjusted intervention variables;

[0062] Based on the second intervention effect, the intervention variable that has the greatest intervention effect on the evaluation index is obtained, and the optimal intervention variable is obtained.

[0063] Based on the optimal intervention variable corresponding to each of the aforementioned causal effect assessment models, an optimized set of hyperparameters is obtained.

[0064] Optionally, in the above-described apparatus, the convergence or termination condition includes at least one of the following: reaching the set number of iterations and satisfying the target value set by the target evaluation index.

[0065] Optionally, the above-mentioned device further includes:

[0066] A building module is used to construct the causal effect assessment model.

[0067] Optionally, in the above-described apparatus, the building module includes:

[0068] The second acquisition submodule is used to acquire the original training datasets and corresponding historical model training data for different scenarios; wherein the historical model training data includes at least hyperparameters and evaluation metrics.

[0069] An extraction submodule is used to extract data metadata features from the original training dataset;

[0070] A generation submodule is used to generate a modeling parameter tuning result table based on the data element features and the historical model training data;

[0071] The model training submodule is used to train the model based on the modeling parameter tuning result table and the causal algorithm to generate a causal effect evaluation model.

[0072] Optionally, in the above apparatus, the model training submodule is specifically used for:

[0073] One hyperparameter is selected from the modeling and parameter tuning result table as an intervention variable, and one evaluation index is selected as the target result. The unselected hyperparameters and data element features in the modeling and parameter tuning result table are used as the set of adjustment variables.

[0074] The causal algorithm is trained based on the intervention variables, the target outcome, and the set of adjustment variables to obtain the causal evaluation model.

[0075] Optionally, in the above apparatus, the extraction submodule is specifically used for:

[0076] Random sampling is performed on the original training dataset to obtain multiple data subsets;

[0077] The meta-feature extractor is used to extract features from the multiple data subsets to obtain multiple meta-feature vectors;

[0078] The average value of the multiple meta-feature vectors is calculated to obtain the data meta-features.

[0079] This invention also provides a model information determination system, characterized in that it includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model information determination method as described above.

[0080] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of determining model information as described above.

[0081] This invention identifies hyperparameters that have a positive impact on evaluation metrics by identifying the causal relationship between hyperparameters and evaluation metrics, thereby improving the accuracy of business model parameter optimization, avoiding extensive computational experiments, reducing model parameter optimization costs, and minimizing the use of system resources during the business model training phase. Attached Figure Description

[0082] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a flowchart of a model information determination method provided in an embodiment of the present invention;

[0084] Figure 2 This is a structural block diagram of a model information determination device provided in an embodiment of the present invention. Detailed Implementation

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] See Figure 1 , Figure 1 This is a flowchart of the model information determination method provided in the embodiments of the present invention.

[0087] like Figure 1 As shown, the method for determining model information includes the following steps:

[0088] Step 101: Obtain the hyperparameters of the target business model to be trained, as well as the data element features and evaluation metrics used to train the target business model.

[0089] Specifically, a training set for training the target business model is obtained, and feature extraction is performed on the training set to obtain the corresponding metadata features. In this process, a target training set for the target business model is obtained, and the Dataset2Vec method is used to extract features from the target training set to obtain the corresponding metadata features.

[0090] To obtain the hyperparameters of the target business model, in this embodiment of the invention, a search algorithm is first applied to search the given search space based on the task type of the target business model and the data element features of the training target business model to obtain a set of hyperparameters. The set of hyperparameters obtained by the search is then used as the hyperparameters of the target business model to be trained.

[0091] Optionally in this embodiment of the invention, obtaining the evaluation indicators set by the user for evaluating the target business model includes the user performing a configuration operation on the settings interface of the model information determination system to determine the target evaluation indicators of the target business model.

[0092] The target business model can be a text recognition model, a speech recognition model, an image processing model, etc.

[0093] Step 102: Obtain the causal relationship between the various hyperparameters of the target business model and the evaluation index.

[0094] The hyperparameters of the target business model and the evaluation metrics exhibit causal relationships of varying strengths. These causal relationships can be obtained using causal discovery algorithms such as the PC algorithm and the No-tears algorithm.

[0095] Step 103: Iteratively adjust each hyperparameter of the target business model according to the data element features, the evaluation index, and the causal relationship to obtain the adjusted target hyperparameter set.

[0096] Specifically, the hyperparameters to be adjusted are determined based on the causal relationship, the causal effect evaluation model corresponding to the evaluation index and the hyperparameters to be adjusted is determined based on the evaluation index, and the causal effect evaluation model is used to iteratively adjust the data element features and the hyperparameters to be adjusted to obtain the adjusted target hyperparameter set.

[0097] The embodiments of the present invention determine the hyperparameters that have a positive impact on the evaluation index by identifying the causal relationship between hyperparameters and evaluation indexes, thereby avoiding a large number of computational experiments, reducing the cost of model parameter optimization, and improving the accuracy of model parameter optimization.

[0098] Optionally, step 103, which iteratively adjusts each hyperparameter of the target business model based on the data element features, the evaluation indicators, and the causal relationships to obtain the adjusted target hyperparameter set, includes:

[0099] Based on the causal relationship, at least one candidate hyperparameter to be adjusted is determined;

[0100] Based on the evaluation index, obtain the causal effect evaluation model that corresponds one-to-one with each of the candidate hyperparameters;

[0101] The data element features and the candidate hyperparameters are processed using the respective causal effect evaluation models to obtain the intervention effect corresponding to each candidate hyperparameter and the evaluation index.

[0102] Based on the effects of each intervention, the candidate hyperparameters are iteratively adjusted.

[0103] Among them, the aforementioned causal effect evaluation models include, but are not limited to, meta-learner models and causal forest models.

[0104] Specifically, users can determine at least one candidate hyperparameter to be adjusted based on the causal relationship between each hyperparameter and the evaluation index. For example, users can identify at least one hyperparameter causally related to the evaluation index as at least one candidate hyperparameter to be adjusted. There is a pre-built causal effect evaluation model between each hyperparameter and the evaluation index. Based on the evaluation index and the identified candidate hyperparameter, a causal effect evaluation model corresponding to each candidate hyperparameter is obtained from the pre-built causal effect evaluation model. This model is then used to evaluate the influence of each candidate hyperparameter on the evaluation index (i.e., the intervention effect of each candidate hyperparameter on the evaluation index), obtaining the intervention effect of each candidate hyperparameter on the evaluation index. This allows the intervention effect of each candidate hyperparameter on the evaluation index to assist in guiding the adjustment of each hyperparameter in the target business model, reducing the number of trials required for adjusting each hyperparameter in the target business model, thereby reducing the cost of model parameter optimization and the resource consumption during the training phase of the target business model.

[0105] Optionally, the step of processing the data meta-features and the candidate hyperparameters using the respective causal effect evaluation models to obtain the intervention effect corresponding to each candidate hyperparameter and the evaluation index includes:

[0106] For each causal effect assessment model, the candidate hyperparameters corresponding to the causal effect assessment model are used as intervention variables, the data element features and other hyperparameters besides the candidate hyperparameters used as intervention variables are used as adjustment variables, and the assessment index is used as the target variable.

[0107] The intervention variable and the adjustment variable are input into the causal effect evaluation model corresponding to the intervention variable to obtain the first intervention effect corresponding to the intervention variable.

[0108] Specifically, when using the causal effect assessment model to process the data element features and each of the candidate hyperparameters, the hyperparameters, data element features, and assessment indicators can be divided into intervention variables, adjustment variables, and target variables. For example, the candidate hyperparameters corresponding to the causal effect assessment model can be used as intervention variables, the data element features and other hyperparameters besides the candidate hyperparameters used as intervention variables can be used as adjustment variables, and the assessment indicators can be used as target variables. The intervention variables and the adjustment variables are input into the causal effect assessment model corresponding to the intervention variables for estimation, so as to obtain different intervention effects corresponding to different intervention variables (i.e., hyperparameters taking different parameter values).

[0109] Optionally, step 103, which iteratively adjusts each hyperparameter of the target business model based on the data element features, the evaluation indicators, and the causal relationships to obtain the adjusted target hyperparameter set, further includes:

[0110] Based on each of the first intervention effects, determine the adjustment method for the intervention variable corresponding to each of the first intervention effects;

[0111] Based on the adjustment method corresponding to each intervention variable, the intervention variables of each causal effect assessment model are adjusted;

[0112] Each of the aforementioned causal effect assessment models was reassessed based on the adjusted intervention variables to obtain an optimized set of hyperparameters;

[0113] The target business model is trained and estimated based on the optimized hyperparameter set and the data element features;

[0114] If the model convergence or termination condition is met, the model training and estimation are terminated, and the optimized hyperparameter set is determined as the target hyperparameter set.

[0115] Specifically, taking a candidate hyperparameter 'a' as an example, a candidate hyperparameter 'a' can have different values ​​from 1 to 4, etc. These values ​​represent different values ​​of the intervention variable, and these different values ​​have different intervention effects on the evaluation index. The intervention effect of these intervention variables on the evaluation index is evaluated based on the causal effect evaluation model, which is also called causal effect estimation. If it is found that the larger the value of the intervention variable, the larger its corresponding causal effect estimate is. That is to say, the larger the value of the candidate hyperparameter 'a' of the target business model, the larger the corresponding evaluation index of the target business model. Therefore, the adjustment method for determining the intervention variable to obtain the intervention value with the maximum causal effect estimate is to increase the value of the intervention variable. Under the guidance of the above-determined adjustment method, the candidate hyperparameter 'a' is continuously optimized and adjusted. Each candidate hyperparameter is adjusted according to the above process to obtain an optimized hyperparameter set. The optimized hyperparameter set is used for model training and estimation on the training dataset of the target business model. If the model converges or the set number of training and evaluation cycles is reached, the model training and estimation ends, and the optimized hyperparameter set is output as the target hyperparameter set, that is, the hyperparameter set with the greatest impact on the target evaluation index is obtained. The convergence or termination conditions include at least one of the following: reaching the set number of iterations and meeting the target value set by the target evaluation index.

[0116] Optionally, the step of re-evaluating the causal effect based on the adjusted intervention variables in each of the causal effect assessment models to obtain an optimized set of hyperparameters includes:

[0117] Each of the aforementioned causal effect assessment models is reassessed based on the adjusted intervention variables to obtain the second intervention effect corresponding to the adjusted intervention variables;

[0118] Based on the second intervention effect, the intervention variable that has the greatest intervention effect on the evaluation index is obtained, and the optimal intervention variable is obtained.

[0119] Based on the optimal intervention variable corresponding to each of the aforementioned causal effect assessment models, an optimized set of hyperparameters is obtained.

[0120] Specifically, taking a candidate hyperparameter as an example, such as hyperparameter 'a', a pre-trained causal effect evaluation model corresponding to the evaluation index is used to evaluate the Individual Treatment Effect (ITE) calculated by different intervention variables (i.e., different parameter values, such as mean and median) under the training samples corresponding to the current data element features. The intervention value with the largest ITE is selected as the optimal intervention value (e.g., mean) for hyperparameter 'a' in this iteration. Following the same method as hyperparameter 'a', the causal effect evaluation of other candidate hyperparameters to be adjusted in the target business model is completed, obtaining the optimal intervention values ​​for different hyperparameters, thereby obtaining the optimized hyperparameter set.

[0121] In this embodiment of the invention, obtaining an optimized hyperparameter set based on the optimal intervention variable corresponding to each causal effect assessment model includes: obtaining an optimal intervention variable set based on the optimal intervention variable corresponding to each causal effect assessment model; selecting at least one target optimal intervention variable from the optimal intervention variable set; adjusting the corresponding hyperparameters in the target business model based on the at least one target optimal intervention variable to obtain optimized hyperparameters; and training and estimating the target business model based on the optimized hyperparameter set and the data element features.

[0122] In one specific embodiment, after obtaining the optimal set of intervention variables based on the optimal intervention variable corresponding to each of the causal effect assessment models, and after completing the causal effect assessment of all hyperparameters, all optimal intervention variables are obtained. They are then sorted according to the order of intervention effect from largest to smallest (or smallest to largest), and the optimal intervention variable with the largest intervention effect is selected as the hyperparameter for this adjustment, thus obtaining the optimized hyperparameter.

[0123] In this embodiment of the invention, a causal effect assessment model between hyperparameters and evaluation indicators is pre-constructed, thereby using the causal effect assessment model to evaluate the intervention effect of different hyperparameter values ​​on the evaluation indicators.

[0124] Optionally, the method further includes:

[0125] Construct the causal effect assessment model.

[0126] The steps for constructing the causal effect assessment model include:

[0127] Obtain the original training datasets and corresponding historical model training data for different scenarios; wherein the historical model training data includes at least hyperparameters and evaluation metrics;

[0128] Extract data metadata features from the original training dataset;

[0129] A modeling parameter tuning result table is generated based on the data element features and the historical model training data.

[0130] Based on the modeling and parameter tuning results table and the causal algorithm, the model is trained to generate a causal effect evaluation model.

[0131] Specifically, a causal estimation model is constructed based on the original training datasets for different scenarios and the corresponding historical model training data. The original training dataset typically consists of multiple rows and columns. Usually, the original training data is a large table, unsuitable for direct modeling, and requires prior processing. In this embodiment, meta-feature extraction is performed on a large number of original training datasets to obtain data meta-features that represent the information contained in the original training datasets; for example, the Dataset2Vec method is used to extract the data meta-features of the original datasets, resulting in a set of vectors representing the information of the original datasets. The obtained data meta-features are integrated with historical model training data, such as the task type of each training dataset, hyperparameters during training, and evaluation metrics for each training iteration, to form a modeling and parameter tuning result table. [f1, f2, ..., fk] in Table 1 represent the data meta-features. Hyperparameters include, but are not limited to, learning rate, regularization parameter, number of neural network layers, and number of training epochs; evaluation metrics include, but are not limited to, accuracy, logloss, and Gini coefficients. Based on the modeling and parameter tuning result table and the causal evaluation algorithm, the model is trained to generate a causal efficacy estimation model. Among them, causal evaluation algorithms include, but are not limited to: Double Machine Learning and Meta-learner algorithms.

[0132] Optionally, the step of extracting data meta-features from the original training dataset includes:

[0133] Random sampling is performed on the original training dataset to obtain multiple data subsets;

[0134] The meta-feature extractor is used to extract features from the multiple data subsets to obtain multiple meta-feature vectors;

[0135] The average value of the multiple meta-feature vectors is calculated to obtain the data meta-features.

[0136] Specifically, the meta-feature extractor in this embodiment of the invention can be a deep neural network-based meta-feature extractor. Before obtaining the meta-feature vectors of the original training dataset, the deep neural network meta-feature extractor is first trained. A mini-batch random sampling method is used to randomly sample certain instance features and their corresponding labels by row and column from the metadata composed of multiple original datasets, forming multiple subset batches. A hierarchical neural network is designed, and this hierarchical neural network is used to train the subset batches, outputting data meta-features. During training, the principle of similarity between similarly distributed meta-features and difference between dissimilarly distributed meta-features can be used to iteratively optimize the distance between the data meta-features obtained from different subset batches after training. This results in a smaller distance between the data meta-features obtained from subset batches trained from the same original dataset and a larger distance between the data meta-features obtained from subset batches trained from different original datasets, thereby optimizing the meta-feature extractor-hierarchical neural network model.

[0137] After optimizing the meta-feature extractor-hierarchical neural network model, mini-batch random sampling is performed on the new original training dataset to obtain multiple data subsets. The hierarchical neural network model is then used to extract features from these subsets, resulting in multiple meta-feature vectors. The average value of these meta-feature vectors is calculated and used as the data meta-feature of the new original training dataset. The elements of the data meta-features are named [f1, f2, ..., fk] and recorded as shown in Table 1 below, which is named the data meta-feature table:

[0138] Table 1 Data element characteristics table

[0139] f1 f2 fk 0.232 0.323 …

[0140] Based on the original training dataset and the business scenario or training task, a search is performed using a grid or random method within the default or user-specified hyperparameter space. The hyperparameters used for each search and the results are recorded in Table 2 below, which is named the search results table.

[0141] Table 2: Search Results Table

[0142]

[0143] The above metadata feature table and search result table are merged, and the data metadata features are filled into each search result to obtain the table shown in Table 2 below, which is recorded as the modeling parameter tuning result table.

[0144] Table 3 Modeling and Parameter Tuning Results

[0145]

[0146] Once enough training is performed on each task, and enough different tasks are completed, a large amount of modeling and hyperparameter tuning data can be accumulated. From this data, the causal relationship between the dataset, hyperparameters, and model-specific metrics can be analyzed.

[0147] Optionally, the step of training the model to generate a causal effect evaluation model based on the modeling parameter tuning result table and the causal algorithm includes:

[0148] One hyperparameter is selected from the modeling and parameter tuning result table as an intervention variable, and one evaluation index is selected as the target result. The unselected hyperparameters and data element features in the modeling and parameter tuning result table are used as the set of adjustment variables.

[0149] The causal algorithm is trained based on the intervention variables, the target outcome, and the set of adjustment variables to obtain the causal evaluation model.

[0150] Specifically, the modeling and parameter tuning result table data obtained in the above embodiments is used to train the causal effect assessment model. The task of causal estimation is to estimate the intervention effect produced after an intervention is applied to a certain unit (research object). In this invention, different hyperparameter values ​​are applied to a certain hyperparameter in a training sample to estimate the intervention effect on the evaluation index. Therefore, the variables in the modeling and parameter tuning result table can be divided into intervention variables, outcome variables (i.e., target variables), and adjustment variables. Among them, the adjustment variables affect the relationship between the independent and dependent variables, including the influence on the direction of the relationship and the influence on the strength of the relationship. As shown in Table 3 above, there are m hyperparameters and n evaluation indicators in the modeling and parameter tuning result table. One hyperparameter is selected as the intervention variable T (treatment), and one evaluation indicator is selected as the target result Y (target_outcome). The hyperparameters and data element features that were not selected in the table data are used as the set of adjustment variables W (adjustmentsets). Using a causal estimation algorithm, such as a dual machine learning algorithm, m*n causal estimation models can be obtained, where each hyperparameter, such as hyperparameter 1, corresponds to n causal estimation models, and each evaluation indicator corresponds to m causal estimation models. Each causal estimation model can estimate the quantitative impact of changes in this hyperparameter on the corresponding evaluation index.

[0151] This invention utilizes a causal estimation model to discover and identify hyperparameters that have a positive impact on the evaluation metrics by using the causal relationship between hyperparameters and model evaluation metrics. This avoids conducting a large number of computational experiments and can guide parameter optimization when performing new model training tasks, thereby reducing the cost of model parameter optimization and improving the accuracy of model parameter optimization.

[0152] Based on the model information determination method provided in the above embodiments, this invention also provides a model information determination apparatus for implementing the above method. Please refer to... Figure 2 This invention provides a model information determination device 200, characterized in that the device 200 includes:

[0153] The first acquisition module 201 is used to acquire the various hyperparameters of the target business model to be trained, as well as the data element features and evaluation metrics used to train the target business model.

[0154] The second acquisition module 202 is used to obtain the causal relationship between the various hyperparameters of the target business model and the evaluation index;

[0155] The iterative adjustment module 203 is used to iteratively adjust each hyperparameter of the target business model according to the data element features, the evaluation index and the causal relationship, so as to obtain the adjusted target hyperparameter set.

[0156] Optionally, the iterative adjustment module 203 includes:

[0157] The first determining submodule is used to determine at least one candidate hyperparameter to be adjusted based on the causal relationship.

[0158] The first acquisition submodule is used to acquire, based on the evaluation index, a causal effect evaluation model corresponding one-to-one with each of the candidate hyperparameters.

[0159] The first causal effect assessment submodule is used to process the data element features and the candidate hyperparameters using the respective causal effect assessment models to obtain the intervention effect corresponding to each candidate hyperparameter and the assessment index.

[0160] The iterative adjustment submodule iteratively adjusts each of the candidate hyperparameters based on the effects of each intervention.

[0161] Optionally, the first causal effect assessment submodule is specifically used for:

[0162] For each causal effect assessment model, the candidate hyperparameters corresponding to the causal effect assessment model are used as intervention variables, the data element features and other hyperparameters besides the candidate hyperparameters used as intervention variables are used as adjustment variables, and the assessment index is used as the target variable.

[0163] The intervention variable and the adjustment variable are input into the causal effect evaluation model corresponding to the intervention variable to obtain the first intervention effect corresponding to the intervention variable.

[0164] Optionally, the iterative adjustment module 203 further includes:

[0165] The second determining submodule is used to determine the adjustment method of the intervention variable corresponding to each of the first intervention effects based on each of the first intervention effects;

[0166] The adjustment submodule is used to adjust the intervention variables of each of the causal effect assessment models based on the adjustment methods corresponding to each intervention variable.

[0167] The second causal effect assessment submodule is used to reassess the causal effect of the adjusted intervention variables using the causal effect assessment model to obtain an optimized hyperparameter set.

[0168] The model training and estimation submodule is used to train and estimate the target business model based on the optimized hyperparameter set and the data element features;

[0169] The third determining submodule is used to terminate model training and estimation if the model convergence or termination conditions are met, and to determine the optimized hyperparameter set as the target hyperparameter set.

[0170] Optionally, the second causal effect assessment submodule is specifically used for:

[0171] Each of the aforementioned causal effect assessment models is reassessed based on the adjusted intervention variables to obtain the second intervention effect corresponding to the adjusted intervention variables;

[0172] Based on the second intervention effect, the intervention variable that has the greatest intervention effect on the evaluation index is obtained, and the optimal intervention variable is obtained.

[0173] Based on the optimal intervention variable corresponding to each of the aforementioned causal effect assessment models, an optimized set of hyperparameters is obtained.

[0174] Optionally, the convergence or termination condition includes at least one of the following: reaching the set number of iterations and satisfying the target value set by the target evaluation index.

[0175] Optionally, the device further includes:

[0176] A building module is used to construct the causal effect assessment model.

[0177] Optionally, the building module includes:

[0178] The second acquisition submodule is used to acquire the original training datasets and corresponding historical model training data for different scenarios; wherein the historical model training data includes at least hyperparameters and evaluation metrics.

[0179] An extraction submodule is used to extract data metadata features from the original training dataset;

[0180] A generation submodule is used to generate a modeling parameter tuning result table based on the data element features and the historical model training data;

[0181] The model training submodule is used to train the model based on the modeling parameter tuning result table and the causal algorithm to generate a causal effect evaluation model.

[0182] Optionally, the model training submodule is specifically used for:

[0183] One hyperparameter is selected from the modeling and parameter tuning result table as an intervention variable, and one evaluation index is selected as the target result. The unselected hyperparameters and data element features in the modeling and parameter tuning result table are used as the set of adjustment variables.

[0184] The causal algorithm is trained based on the intervention variables, the target outcome, and the set of adjustment variables to obtain the causal evaluation model.

[0185] Optionally, the extraction submodule is specifically used for:

[0186] Random sampling is performed on the original training dataset to obtain multiple data subsets;

[0187] The meta-feature extractor is used to extract features from the multiple data subsets to obtain multiple meta-feature vectors;

[0188] The average value of the multiple meta-feature vectors is calculated to obtain the data meta-features.

[0189] This invention provides a model parameter processing system, characterized in that it includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the model information determination method as described in the above embodiments.

[0190] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the model information determination method as described in the above embodiments.

[0191] This invention also provides a readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described model information determination method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0192] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0194] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A model information determination method characterized by comprising: The method comprises: obtaining respective hyperparameters of a target business model to be trained, and data element features and evaluation indexes for training the target business model; obtaining a causal relationship between the respective hyperparameters of the target business model and the evaluation indexes; wherein the respective hyperparameters of the target business model and the evaluation indexes have different strong and weak causal relationships; iteratively adjusting the respective hyperparameters of the target business model according to the data element features, the evaluation indexes and the causal relationship to obtain an adjusted target hyperparameter set; the step of iteratively adjusting the respective hyperparameters of the target business model according to the data element features, the evaluation indexes and the causal relationship to obtain an adjusted target hyperparameter set comprises: determining at least one candidate hyperparameter to be adjusted according to the causal relationship; obtaining a causal effect evaluation model corresponding to each of the candidate hyperparameters based on the evaluation indexes; processing the data element features and each of the candidate hyperparameters using each of the causal effect evaluation models to obtain an intervention effect corresponding to each of the candidate hyperparameters and the evaluation indexes; iteratively adjusting each of the candidate hyperparameters based on each of the intervention effects.

2. The model information determination method according to claim 1, characterized by, the step of processing the data element features and each of the candidate hyperparameters using each of the causal effect evaluation models to obtain an intervention effect corresponding to each of the candidate hyperparameters and the evaluation indexes comprises: for each causal effect evaluation model, taking the candidate hyperparameter corresponding to the causal effect evaluation model as an intervention variable, taking the data element features and other hyperparameters except the candidate hyperparameter as adjustment variables, and taking the evaluation indexes as a target variable; inputting the intervention variable and the adjustment variables into the causal effect evaluation model corresponding to the intervention variable to obtain a first intervention effect corresponding to the intervention variable.

3. The model information determination method according to claim 2, characterized by, the step of iteratively adjusting the respective hyperparameters of the target business model according to the data element features, the evaluation indexes and the causal relationship to obtain an adjusted target hyperparameter set comprises: determining an adjustment mode of the intervention variable corresponding to each of the first intervention effects according to each of the first intervention effects; adjusting the intervention variable of each of the causal effect evaluation models based on the adjustment mode corresponding to each intervention variable; each of the causal effect evaluation models re-evaluates the causal effect based on the adjusted intervention variable to obtain an optimized hyperparameter set; and the target business model is trained and estimated based on the optimized hyperparameter set and the data element features; if an end condition is met, the model training and estimation are ended, and the optimized hyperparameter set is determined as the target hyperparameter set.

4. The model information determination method according to claim 3, characterized by, the step of each of the causal effect evaluation models re-evaluating the causal effect based on the adjusted intervention variable to obtain an optimized hyperparameter set comprises: each of the causal effect evaluation models re-evaluates the causal effect based on the adjusted intervention variable to obtain a second intervention effect corresponding to the adjusted intervention variable. According to the second intervention effect, an intervention variable with the maximum intervention effect on the evaluation index is obtained, and an optimal intervention variable is obtained; According to the optimal intervention variable corresponding to each causal effect evaluation model, an optimized hyperparameter set is obtained.

5. The model information determination method according to claim 3, characterized by, The end condition includes at least one of the following: reaching a set number of iterations and meeting a target evaluation index set target value.

6. The model information determination method according to any one of claims 1 to 5, characterized by, The method further includes constructing the causal effect evaluation model by the following steps: Obtain an original training data set and historical model training data of different scenarios; wherein the historical model training data at least includes hyperparameters and evaluation indexes; Extract data element features from the original training data set; Generate a modeling parameter tuning result table based on the data element features and the historical model training data; Generate a causal effect evaluation model according to the modeling parameter tuning result table and a causal algorithm.

7. The model information determination method according to claim 6, characterized by, The step of generating a causal effect evaluation model according to the modeling parameter tuning result table and a causal algorithm includes: Select a hyperparameter from the modeling parameter tuning result table as an intervention variable, and select an evaluation index as a target result, and select the unselected hyperparameters and data element features in the modeling parameter tuning result table as an adjustment variable set; Train the causal algorithm based on the intervention variable, the target result, and the adjustment variable set to obtain the causal effect evaluation model.

8. The model information determination method according to claim 6, characterized by, The step of extracting data element features from the original training data set includes: Randomly sample the original training data set to obtain a plurality of data subsets; Use a meta-feature extractor to extract features from the plurality of data subsets to obtain a plurality of meta-feature vectors; Determine the average value of the plurality of meta-feature vectors as the data element features.

9. A model information determining apparatus characterized by comprising: The device includes: A first obtaining module is configured to obtain each hyperparameter of a target business model to be trained, and a data element feature and an evaluation index used for training the target business model; A second obtaining module is configured to obtain a causal relationship between each hyperparameter of the target business model and the evaluation index; wherein the causal relationship between each hyperparameter of the target business model and the evaluation index has different strengths; An iterative adjustment module is configured to iteratively adjust each hyperparameter of the target business model according to the data element feature, the evaluation index, and the causal relationship to obtain an adjusted target hyperparameter set; The iterative adjustment module includes: A first determining submodule is configured to determine at least one candidate hyperparameter to be adjusted according to the causal relationship; A first obtaining submodule is configured to obtain a causal effect evaluation model corresponding to each candidate hyperparameter based on the evaluation index; A first causal effect evaluation submodule is configured to process the data element feature and each candidate hyperparameter using each causal effect evaluation model to obtain an intervention effect corresponding to each candidate hyperparameter and the evaluation index; An iterative adjustment submodule is configured to iteratively adjust each candidate hyperparameter based on each intervention effect.

10. The model information determination apparatus according to claim 9, characterized by, The first causal effect evaluation submodule is specifically configured to: For each causal effect evaluation model, a candidate hyperparameter corresponding to the causal effect evaluation model is taken as an intervention variable, the data element feature and other hyperparameters except the candidate hyperparameter taken as the intervention variable are taken as adjustment variables, and the evaluation index is taken as a target variable; The intervention variable and the adjustment variable are input into the causal effect evaluation model corresponding to the intervention variable to obtain a first intervention effect corresponding to the intervention variable.

11. The model information determination apparatus according to claim 10, characterized by, The iterative adjustment module further includes: A second determination submodule is configured to determine an adjustment mode of the intervention variable corresponding to each first intervention effect according to the first intervention effect; An adjustment submodule is configured to adjust the intervention variable of each causal effect evaluation model based on the adjustment mode corresponding to each intervention variable; A second causal effect evaluation submodule is configured to perform causal effect evaluation again based on the adjusted intervention variable for each causal effect evaluation model to obtain an optimized hyperparameter set; A model training and estimation submodule is configured to perform model training and estimation on the target business model based on the optimized hyperparameter set and the data element feature; A third determination submodule is configured to end the model training and estimation and determine the optimized hyperparameter set as a target hyperparameter set if an end condition is met.

12. The model information determination apparatus according to claim 11, characterized by, The second causal effect evaluation submodule is specifically configured to: Each causal effect evaluation model performs causal effect evaluation again based on the adjusted intervention variable to obtain a second intervention effect corresponding to the adjusted intervention variable; An intervention variable with the greatest intervention effect on the evaluation index is obtained based on the second intervention effect to obtain an optimal intervention variable; An optimized hyperparameter set is obtained based on the optimal intervention variable corresponding to each causal effect evaluation model.

13. The model information determination apparatus according to claim 11, characterized by, The end condition includes at least one of the following: reaching a set iteration number and meeting a target evaluation index set target value.

14. The model information determination apparatus according to any one of claims 9 to 13, characterized by, The device includes: A construction module configured to construct the causal effect evaluation model; The construction module includes: A second acquisition submodule is configured to acquire original training data sets and historical model training data of different scenarios; the historical model training data at least includes hyperparameters and evaluation indexes; An extraction submodule is configured to extract data element features from the original training data sets; A generation submodule is configured to generate a modeling parameter adjustment result table based on the data element features and the historical model training data; A model training submodule is configured to generate a causal effect evaluation model based on the modeling parameter adjustment result table and a causal algorithm.

15. The model information determination apparatus according to claim 14, characterized by, The model training submodule is specifically configured to: Select a hyperparameter from the modeling parameter adjustment result table as an intervention variable and select an evaluation index as a target result in sequence, and take unselected hyperparameters and data element features in the modeling parameter adjustment result table as an adjustment variable set; Train a causal algorithm based on the intervention variable, the target result, and the adjustment variable set to obtain the causal effect evaluation model.

16. The model information determination apparatus according to claim 14, characterized by, The extraction submodule is specifically configured to: Randomly sample the original training data sets to obtain a plurality of data subsets; extracting features of the plurality of data subsets using a meta-feature extractor to obtain a plurality of meta-feature vectors; determining an average of the plurality of meta-feature vectors as the data meta-feature.

17. A model information determining system characterized by comprising: A processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model information determining method according to any one of claims 1 to 8. 18.A computer readable storage medium, storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the model information determining method according to any one of claims 1 to 8.

Images