Method and system for automatically training machine learning models

Abstract

Disclosed are a method and system for automatically training a machine learning model. A method for automatically training a machine learning model performed by a computing device includes: obtaining a data set comprising a plurality of data records; determining a feature engineering strategy for the data set by meta-learning; generating features related to each data record in the data set using the determined feature engineering strategy; constructing a machine learning sample based at least on the generated features; and training a machine learning model using the constructed machine learning sample based on automatically obtained hyperparameter values.

Description

Technical Field

[0001] This application generally relates to the field of artificial intelligence, and more specifically, to a method and system for automatically training machine learning models. Background Technology

[0002] With the emergence of massive amounts of data, artificial intelligence (AI) technology has developed rapidly. However, to extract value from this vast amount of data, personnel not only need to be proficient in AI technologies (especially machine learning), but also need to be very familiar with the specific scenarios in which machine learning is applied (e.g., image processing, speech processing, text processing, and business decision-making areas such as automatic control, financial transactions, and internet advertising). For example, in machine learning, model training typically requires the participation of numerous machine learning experts (including feature engineering strategy selection and model hyperparameter tuning during training). Furthermore, the performance of the trained model often depends on the machine learning experts' personal experience in model training and their familiarity with the specific application scenarios or related business. If the personnel lack sufficient understanding of the specific business or have insufficient modeling experience, it can easily lead to poor model performance. Therefore, existing model training methods not only consume a significant amount of human resources but also result in a high barrier to entry for machine learning. Moreover, due to the reliance on the personal experience and business familiarity of the personnel, the trained model may not objectively reflect real business patterns, all of which pose significant challenges to the industrial application prospects of machine learning technology. Therefore, technical personnel hope to adopt technological means to lower the barriers to model training and application while ensuring the effectiveness of model training. Summary of the Invention

[0003] According to an exemplary embodiment of this application, a method for automatically training a machine learning model, executed by a computing device, is provided. The method may include: acquiring a dataset comprising multiple data records; determining a feature engineering strategy for the dataset through meta-learning; generating features related to each data record in the dataset using the determined feature engineering strategy; constructing machine learning samples based at least on the generated features; and training a machine learning model using the constructed machine learning samples based on automatically obtained hyperparameter values.

[0004] Optionally, the step of determining a feature engineering strategy for the dataset through meta-learning may include: analyzing the dataset to obtain meta-features that reflect the overall characteristics of the dataset; and determining a feature engineering strategy for the dataset based on the obtained meta-features.

[0005] Optionally, the step of determining a feature engineering strategy for the dataset based on the obtained meta-features may include: automatically determining a feature engineering strategy for the dataset based on the obtained meta-features based on the correspondence between the meta-features and the feature engineering strategy; or determining a feature engineering strategy for the dataset using another machine learning model based on the obtained meta-features.

[0006] Optionally, the step of using another machine learning model to determine the feature engineering strategy for the dataset may include: using the obtained meta-features and a clustering model (which is the other machine learning model) to determine historical datasets similar to the dataset that already have corresponding model training results, and determining the feature engineering strategy corresponding to the historical dataset as the feature engineering strategy for the dataset; or using a prediction sample constructed based on the obtained meta-features as the prediction model of the other machine learning model, and using the prediction result of the prediction model for the prediction sample as the feature engineering strategy for the dataset, wherein the prediction model is pre-trained to predict the feature engineering strategy corresponding to the meta-features based on the meta-features.

[0007] Optionally, the step of analyzing the dataset to obtain meta-features that reflect the overall characteristics of the dataset may include: analyzing the dataset to obtain statistical information related to the number of data records included in the dataset, the types of features included in the data records, the number of features and / or the values ​​of the features, as the meta-features.

[0008] Optionally, the meta-features may include at least one of the following: the number of data records included in the dataset, the number of discrete features included in the data records, the number of continuous features included in the data records, whether there are time features in the data records, the number of time features included in the data records, the fill factor of each discrete feature, the fill factor of each continuous feature, the average fill factor of all discrete features and the average fill factor of all continuous features, the mean, variance, median, maximum or minimum value of each continuous feature, or the number of different values ​​of each discrete feature, or the result obtained by combining and statistically analyzing the above items.

[0009] Optionally, the step of determining a feature engineering strategy for the dataset based on the obtained meta-features may include at least one of the following operations: in response to the distribution that discrete features are stronger than continuous features, the feature engineering strategy for the dataset is determined as follows: discretizing the continuous features to obtain discretized features, and searching for a combination of discrete features beneficial to the model training effect in the Cartesian product combination of the original discrete features and the discretized features, as the generated features related to each data record; in response to the distribution that continuous features are stronger than discrete features, the feature engineering strategy for the dataset is determined as follows: training a gradient boosting decision tree model using the dataset, and assigning each decision tree to the sample formed based on each data record. The leaf node number is used as the generated feature related to each data record; in response to the presence of time features in the dataset, the feature engineering strategy for the dataset is determined as follows: the multiple data records are sorted according to the time features, and during the process of training the logistic regression model sequentially using the sorted data records, the time-series features corresponding to the current data record are obtained based on the parameter terms related to the model parameters of the logistic regression model, and used as the generated feature related to each data record; in response to the inclusion of multiple time features in the dataset, the feature engineering strategy for the dataset is determined as follows: for each data record, the feature related to each data record is generated by subtracting the multiple time features in pairs.

[0010] Optionally, the step of training a machine learning model using the constructed machine learning samples based on automatically obtained hyperparameter values ​​may include: using a portion of all constructed machine learning samples as a training sample set; automatically generating multiple sets of hyperparameter values ​​that affect the training of the machine learning model by adopting different hyperparameter generation strategies according to the number of training samples included in the training sample set; training multiple machine learning models using the constructed machine learning samples based on each set of hyperparameter values; selecting the top-performing machine learning models from the trained machine learning models and fusing the selected machine learning models through a predetermined model fusion strategy to determine the final trained machine learning model.

[0011] Optionally, the step of automatically generating multiple sets of hyperparameter values ​​that affect the training of the machine learning model by adopting different hyperparameter generation strategies according to the number of training samples included in the training sample set may include: in response to the number of training samples included in the training sample set being less than or equal to a preset threshold, generating multiple sets of hyperparameter values ​​in the following ways: randomly generating multiple sets of hyperparameter values ​​within the range of hyperparameter values ​​corresponding to the number of training samples; or, predefining the hyperparameter values ​​of some hyperparameters in the hyperparameter set, and randomly generating the remaining hyperparameter values ​​within the range of hyperparameter values ​​corresponding to the number of training samples. Multiple sets of hyperparameter values ​​for the remaining hyperparameters; in response to the number of training samples included in the training sample set being greater than the preset threshold, multiple sets of hyperparameter values ​​are generated in the following manner: predefine the hyperparameter values ​​of some hyperparameters in the hyperparameters, and for the remaining hyperparameters, generate multiple sets of hyperparameter values ​​by combining a method of randomly generating hyperparameter values ​​within the range of hyperparameter values ​​corresponding to the number of training samples and a method of generating hyperparameter values ​​using a Bayesian optimization model, wherein the Bayesian optimization model is constructed to reflect the relationship between the hyperparameter values ​​and their corresponding model effects.

[0012] Optionally, the step of generating multiple sets of hyperparameter values ​​for the remaining hyperparameters according to the combination method may include: constructing a sub-training sample set by sampling training samples in the training sample set; randomly generating a predetermined set of initial hyperparameter values ​​for the remaining hyperparameters within a range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set; training a machine learning model using the sub-training sample set based on each set of hyperparameter values ​​consisting of predefined partial hyperparameter values ​​and each set of initial hyperparameter values ​​for the remaining hyperparameters, and validating the model performance of the trained machine learning model to obtain the model performance corresponding to each set of hyperparameter values; establishing a Bayesian optimization model based on each set of hyperparameter values ​​and its corresponding model performance; continuously determining the number of sets of hyperparameter values ​​that still need to be generated for the remaining hyperparameters, generating a portion of this number using the Bayesian optimization model, and randomly generating another portion of this number within a range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set.

[0013] Optionally, the step of sampling training samples in the training sample set to construct a sub-training sample set may include: sampling training samples in the training sample set multiple times to construct multiple sub-training sample sets with different numbers of training samples, wherein in the multiple samplings, the sub-training sample set obtained by the later sampling includes more training samples than the previous sampling.

[0014] Optionally, the step of generating multiple sets of hyperparameter values ​​for the remaining hyperparameters according to the combination method may further include: whenever new hyperparameter values ​​for the remaining hyperparameters are generated according to the combination method, training a machine learning model using a sub-training sample set obtained from the first sampling based on each set of hyperparameter values ​​composed of the predefined hyperparameter values ​​of the partial hyperparameters and the generated new hyperparameter values; selecting the multiple sets of hyperparameter values ​​with the highest corresponding model performance among all sets of hyperparameter values ​​used when training the machine learning model using the sub-training sample set obtained from the previous sampling, and using them as all sets of hyperparameter values ​​to be used when training the machine learning model using the sub-training sample set obtained from the next sampling; stopping sampling in response to a predetermined condition being met, and using the multiple sets of hyperparameter values ​​with the highest corresponding model performance among all sets of hyperparameter values ​​used when training the machine learning model using the sub-training sample set obtained from the last sampling as the multiple sets of hyperparameter values ​​affecting the training of the machine learning model.

[0015] Optionally, the step of constructing a sub-training sample set may include: constructing a sub-training sample set by performing density-based subsampling on the training samples in the training sample set.

[0016] Optionally, the step of constructing a sub-training sample set by performing density-based subsampling on the training samples in the training sample set may include: reducing the training samples in the training sample set to another dimension space; and sampling the training samples according to the labels of the training samples and the distribution density of the training samples in the other dimension space to construct the sub-training sample set.

[0017] Optionally, the step of reducing the dimensionality of the training sample set to the other dimensional space may include: mapping each training sample in the training sample set to a sample point on a two-dimensional plane. Optionally, the step of sampling the training samples to construct a sub-training sample set based on the label of the training samples and the distribution density of the training samples in the other dimensional space may include: classifying all sample points on the two-dimensional plane into a positive sample point set and a negative sample point set according to whether the training samples are positive or negative samples; for each point set in the positive sample point set and the negative sample point set, sampling the sample points according to the distribution density of the sample points in the point set according to a set sampling ratio, and taking the set of training samples corresponding to the sampled sample points as the sub-training sample set.

[0018] Optionally, the step of sampling sample points according to the distribution density of sample points in the point set according to a set sampling ratio may include: determining a planar region covering all sample points in the point set based on the maximum and minimum values ​​of the abscissa and ordinate of all sample points in the point set on a two-dimensional plane; dividing the determined planar region into multiple sub-regions on an average basis; counting the number of sub-regions in which sample points fall; in response to the number of sub-regions in which sample points fall being greater than a preset number, randomly sampling in each sub-region according to a set sampling ratio; in response to the number of sub-regions in which sample points fall being less than or equal to the preset number, increasing the number of sub-regions until the number of sub-regions in which sample points fall being greater than the preset number, and then randomly sampling in each sub-region according to a set sampling ratio.

[0019] Optionally, the step of generating multiple sets of hyperparameter values ​​according to the combination method for the remaining hyperparameters may further include: after the Bayesian optimization model is established, whenever a new hyperparameter value and its corresponding model effect are obtained, the Bayesian optimization model is updated using it.

[0020] Optionally, if the number of training samples in the training sample set is greater than the preset threshold, the step of determining the final trained machine learning model may include: obtaining several sets of hyperparameter values ​​used when training the top-ranked machine learning models; performing secondary training on the machine learning model using the training sample set based on each set of hyperparameter values; and fusing the secondary trained machine learning models using a predetermined model fusion strategy to determine the final trained machine learning model.

[0021] Optionally, the step of determining the final trained machine learning model may include: using the plurality of machine learning models as an initial model candidate set; performing the following model combination operation on the initial model candidate set: combining the machine learning model with the best performance among the plurality of machine learning models with the machine learning models in the initial model candidate set, and calculating the model performance of each combined machine learning model group; determining whether the model performance of each combined machine learning model group is improved compared to the best model performance before combination; in response that the model performance of the combined machine learning model group is not improved compared to the best model performance before combination, determining the machine learning model with the best performance before combination as the final machine learning model; in response that the model performance of the combined machine learning model group is improved compared to the best model performance before combination, adding the combined machine learning model group with the best performance after combination to the initial model selection set to constitute the current model candidate set; repeating the same model combination operation as the one performed on the initial model candidate set for the current model candidate set a predetermined number of times, and selecting the machine learning model combination with the best performance in the final generated current model candidate set; determining the weights of the plurality of machine learning models in model fusion based on the number of each machine learning model included in the plurality of machine learning models in the machine learning model combination.

[0022] Optionally, the step of determining the final trained machine learning model may further include: obtaining several sets of hyperparameter values ​​used when acquiring the top-ranked machine learning models, and the number of rounds to train the machine learning model using the training sample set based on each set of hyperparameter values; training the machine learning model for the corresponding number of rounds using all constructed machine learning samples based on each set of hyperparameter values ​​to obtain several retrained machine learning models; and fusing the several retrained machine learning models according to their respective weights determined during model fusion to obtain the final trained machine learning model.

[0023] Optionally, the machine learning model may be a neural network model or a logistic regression model.

[0024] Optionally, the operation of generating a portion of the set of numbers using a Bayesian optimization model and the operation of randomly generating another portion of the set of numbers can be performed in parallel.

[0025] Optionally, the dataset may include image data, voice data, text data, and / or business data.

[0026] According to another exemplary embodiment of this application, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the above-described method.

[0027] According to another exemplary embodiment of this application, a system is provided that includes at least one computing device and at least one storage device containing storage instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the method described above.

[0028] According to another exemplary embodiment of this application, a system for automatically training a machine learning model is provided. The system may include: a dataset acquisition device configured to acquire a dataset comprising multiple data records; a feature engineering strategy determination device configured to determine a feature engineering strategy for the dataset through meta-learning; a feature generation device configured to generate features related to each data record in the dataset using the determined feature engineering strategy; a sample construction device configured to construct machine learning samples based at least on the generated features; and a training device configured to train a machine learning model using the constructed machine learning samples based on automatically obtained hyperparameter values.

[0029] The method and system for automatically training machine learning models according to exemplary embodiments of this application can determine feature engineering strategies through meta-learning, generate sample features using the determined feature engineering strategies to construct machine learning samples, and train machine learning models using automatically obtained hyperparameters with the constructed machine learning samples. This reduces the dependence of model training on the personal experience of machine learning experts and eliminates the need for relevant business personnel to have in-depth understanding of the business. It not only lowers the threshold for machine learning applications but also improves model training efficiency while ensuring training effectiveness. Attached Figure Description

[0030] These and / or other aspects and advantages of this application will become clearer and more readily understood from the following detailed description of embodiments of this application taken in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 A block diagram of a system for automatically training a machine learning model is shown according to an exemplary embodiment;

[0032] Figure 2 This diagram illustrates the generation of features using a gradient boosting decision tree model according to an exemplary embodiment.

[0033] Figure 3 A flowchart illustrating a method for automatically training a machine learning model according to an exemplary embodiment;

[0034] Figure 4A flowchart illustrating an example of a method for automatically training a machine learning model according to an exemplary embodiment. Detailed Implementation

[0035] To enable those skilled in the art to better understand this application, exemplary embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementations.

[0036] Figure 1 This is a block diagram illustrating a system 100 for automatically training machine learning models according to an exemplary embodiment (hereinafter, for ease of description, it is simply referred to as a "model automatic training system"). Figure 1 As shown, the automatic model training system 100 may include a dataset acquisition device 110, a feature engineering strategy determination device 120, a feature generation device 130, a sample construction device 140, and a training device 150.

[0037] Specifically, the dataset acquisition device 110 can acquire a dataset comprising multiple data records. Here, the dataset may involve image data, voice data, text data, and / or business data, but is not limited to these. For example, a data record may involve various information or its transformations used to characterize an image signal; or, a data record may involve various information or its transformations used to characterize a voice signal; or, a data record may involve various components or their transformations used to characterize text. As another example, the data record may involve various attributes related to real-world business operations (e.g., marketing, advertising, recommendation, risk control, automatic control, etc.). In fact, the dataset can involve various data objectively generated in any business scenario, such as data related to user clicks on products in an online marketplace, or data related to user loan applications at a bank. Furthermore, each data record in the dataset may include multiple data attribute fields (e.g., user's age, occupation, personal preferences, login time, product price, color, etc.), and as an example, each business data record may be a form of data. Here, each data record may include discrete features (e.g., occupation = teacher) and / or continuous features (e.g., age = 17), and the data record may involve time features (e.g., time of login to the application = 8:30 a.m.) and / or non-time features.

[0038] Furthermore, the data records in the aforementioned dataset can originate directly from data acquisition devices such as image capturing devices or microphones, various sensors, and internet media. They can also originate from the same or different data sources, such as data providers, the internet (e.g., social networking sites), mobile operators, app operators, courier companies, and credit agencies. Multiple data records from different data sources can be combined to form a data record containing more information. For example, in addition to acquiring the information data records filled out by a customer when applying for a credit card at a bank (including attribute information fields such as income, education, job title, and asset status), the data record acquisition device 100 can also acquire other data records of the customer at the bank, such as loan records and daily transaction data. These acquired data records can be further combined into a complete data record. Alternatively, the data record acquisition device 110 can combine multiple forms containing different data records into a wide table form through their shared attribute fields, serving as the aforementioned dataset (where one form data record in the wide table form corresponds to one data record in the aforementioned dataset).

[0039] As an example, the data recording acquisition device 100 can acquire data manually, semi-automatically, or fully automatically, or process the acquired data to give the processed data records an appropriate format or form (e.g., in the form of form data records). Here, the data recording acquisition device 100 can receive data records manually entered by the user through an input device (e.g., a workstation), or the data recording acquisition device 100 can systematically retrieve data records from the data source in a fully automatic manner, for example, by systematically requesting the data source and obtaining the requested data from the response through a timer mechanism implemented in software, firmware, hardware, or a combination thereof, or it can automatically acquire data with human intervention, for example, by requesting data acquisition upon receiving specific user input. Each time data is acquired, preferably, the data recording acquisition device 110 can store the captured data in non-volatile memory. As an example, a data warehouse can be used to store the acquired raw data and the processed data records.

[0040] However, it should be understood that this application does not impose any restrictions on the type, format, acquisition method, or source of the dataset.

[0041] The feature engineering strategy determination device 120 can determine a feature engineering strategy for the dataset through meta-learning. Typically, different datasets may have different characteristics (e.g., different feature distributions), and selecting an appropriate feature engineering strategy for datasets with different characteristics can better extract the learning-friendly experiential information contained in the dataset. For example, this information can aid business decision-making. Here, meta-learning can be a process of learning from the dataset (e.g., statistical analysis, machine learning, etc.) to obtain a feature engineering strategy suitable for that dataset. For example, the feature engineering strategy determination device 120 can analyze the dataset to obtain meta-features reflecting the overall characteristics of the dataset, and determine a feature engineering strategy for the dataset based on the obtained meta-features. Here, meta-features are features reflecting the overall characteristics of the dataset, rather than features reflecting the characteristics of individual data records in the dataset; therefore, they are different from the discrete or continuous features included in each data record. As an example, the feature engineering strategy determination device 120 can analyze the dataset to obtain statistical information related to the number of data records included in the dataset, the types of features included in the data records, the number of features, and / or the values ​​of the features, as meta-features. Here, for example, statistical information related to the type of features included in a data record could be whether the features included in the data record are discrete or continuous, whether time features exist in the data record, or whether multiple time features exist in the data record. Statistical information related to the number of features included in a data record could be the number of discrete features, the number of continuous features, the number of time features, a comparison between the number of discrete features and the number of continuous features (e.g., data with more discrete features than continuous features, or more continuous features than discrete features), but is not limited to these. Statistical information related to the values ​​of the features included in a data record could be the number of discrete and continuous features with values, the number of discrete and continuous features without values, or the ratio of the number of discrete features with values ​​to the total number of discrete features, but is not limited to these.

[0042] According to an exemplary embodiment of this application, the meta-feature may include at least one of the following: the number of data records included in the dataset, the number of discrete features included in the data records, the number of continuous features included in the data records, whether a time feature exists in the data records, the number of time features included in the data records, the fill factor of each discrete feature, the fill factor of each continuous feature, the average fill factor of all discrete features and the average fill factor of all continuous features, the mean, variance, median, maximum or minimum value of each continuous feature, or the number of different values ​​of each discrete feature, or the result obtained by combining and statistically analyzing the above items, but not limited thereto, as long as it can reflect the overall characteristics of the dataset. Here, the fill factor of each discrete feature may refer to the ratio of data records with that discrete feature value to all data records in the dataset. The fill factor of each continuous feature may refer to the ratio of data records with that continuous feature value to all data records in the dataset. The average fill factor of all discrete features may refer to the average fill factor of all discrete features included in the data records. The average fill factor of all continuous features may refer to the average fill factor of all continuous features included in the data records.

[0043] According to an exemplary embodiment, the feature engineering strategy determination device 120 can automatically determine a feature engineering strategy for the dataset based on the obtained meta-features, according to the correspondence between meta-features and feature engineering strategies. For example, through extensive engineering analysis and experiments, the correspondence between different meta-features and their corresponding feature engineering strategies can be pre-established. Therefore, after obtaining meta-features reflecting the overall characteristics of the dataset through analysis, the feature engineering strategy corresponding to the obtained meta-features can be automatically determined based on the pre-established correspondence between meta-features and feature engineering strategies, and used as the feature engineering strategy for the dataset.

[0044] Alternatively, according to another exemplary embodiment, the feature engineering strategy determination device 120 may determine a feature engineering strategy for the dataset based on the obtained meta-features using another machine learning model. For example, the other machine learning model may be a clustering model or other prediction model pre-trained to perform predictions, but is not limited thereto.

[0045] Specifically, as an example, the feature engineering strategy determination device 120 can use a clustering model based on the obtained meta-features to determine historical datasets that are similar to the dataset acquired by the dataset acquisition device 110 and already have corresponding model training results, and determine the feature engineering strategy corresponding to the historical dataset as the feature engineering strategy for the dataset. That is, the feature engineering strategy determination device 120 can use a clustering model to determine historical datasets similar to the acquired dataset based on the meta-features of the dataset (for example, if the meta-features of two datasets are similar, then the two datasets can be considered similar). If the performance of a machine learning model previously trained using the historical dataset has been verified to meet predetermined conditions, then the feature engineering strategy historically used for that historical dataset can be determined as the feature engineering strategy for the dataset. Here, the clustering model can be, for example, K-means, KNN, etc., but is not limited to these.

[0046] As another example, the feature engineering strategy determination device 120 can input a prediction sample based on the obtained meta-features as the prediction model of the other machine learning model, and use the prediction result of the prediction model for the prediction sample as the feature engineering strategy for the dataset. Here, the prediction model can be pre-trained to predict the feature engineering strategy corresponding to the meta-features based on the meta-features. For example, the other machine learning model can be pre-trained by constructing training samples based on the meta-features and using the feature engineering strategy corresponding to the meta-features as the category label of the training samples, and then using the constructed training samples to train the other machine learning model. Afterwards, the trained other machine learning model can be used as a prediction model to perform predictions on the prediction samples constructed based on the obtained meta-features to provide corresponding prediction results, wherein the prediction results can directly indicate the feature engineering strategy corresponding to the obtained meta-features.

[0047] According to an exemplary embodiment, when determining a feature engineering strategy based on the obtained meta-features, the feature engineering strategy determination device 120 may, in response to the condition that discrete features are stronger than continuous features in distribution, determine the feature engineering strategy for the dataset as follows: discretize the continuous features to obtain discretized features, and search for a combination of discrete features beneficial to the model training effect in the Cartesian product combination of the original discrete features and the discretized features, as the generated features related to each data record. Here, "discrete features are stronger than continuous features in distribution" can be meta-features that reflect the overall characteristics of the dataset. For example, if the number of discrete features is greater than the number of continuous features or the average fill factor of all discrete features is greater than the average fill factor of all continuous features, then discrete features can be considered stronger than continuous features in distribution; however, the criteria for measuring whether discrete features are stronger than continuous features in distribution are not limited to the above methods. As an example, when the analyzed meta-features indicate that discrete features are stronger than continuous features in distribution, the feature engineering strategy determination device 120 can discretize the continuous features included in the data record to obtain new discrete features (e.g., continuous features can be discretized by binning to obtain discretized features). Based on samples including at least the original discrete features from each data record and the obtained new discrete features, a logistic regression model is trained. The device then searches for combinations of discrete features beneficial to the model training effect of the logistic regression model within the Cartesian product of the original and discretized features, as the generated features related to each data record. For example, if the machine learning model to be automatically trained in this application is a neural network model, and each layer of the neural network model is a logistic regression model, then combinations of discrete features beneficial to the model training effect of the logistic regression model will also be beneficial to the model training effect of the neural network model. Therefore, such combinations of discrete features can be generated as features included in the training samples used to train the neural network model. Here, when searching for discrete feature combinations beneficial to the training effect of the logistic regression model from the Cartesian product of the original discrete features and the discretized features, the combined features can be added to the original training samples to train the logistic regression model. Then, it is determined whether the model performance of the logistic regression model trained with the added combined features is improved compared to the model trained with the unadded combined features. If there is an improvement, the combined features are considered beneficial to the model training effect and can be used as the generated features related to each data record. It should be noted that the above method of generating discrete feature combinations through machine learning is only an example; combined features can also be generated through other non-machine learning methods. In addition, the model used in the method of generating discrete feature combinations according to machine learning is not limited to the logistic regression model, but can be any other machine learning model.

[0048] Alternatively, in response to the distributional strength of continuous features over discrete features, the feature engineering strategy determination device 120 may determine the feature engineering strategy for the dataset as follows: train a gradient boosting decision tree model using the dataset, and use the leaf node number of each decision tree to which the samples formed based on each data record fall as the generated feature related to each data record. Here, the distributional strength of continuous features over discrete features may be a meta-feature reflecting the overall characteristics of the dataset. For example, if the number of continuous features is greater than the number of discrete features, or the average fill factor of all continuous features is greater than the average fill factor of all discrete features, then continuous features may be considered to be distributionally stronger than discrete features; however, the criteria for measuring the distributional strength of continuous features over discrete features are not limited to the above methods. When training a gradient boosting decision tree model using the dataset, training samples may be constructed based on the continuous features included in each data record, or training samples may be constructed based on the continuous features included in each data record along with a small number of discrete features. Here, a gradient boosting decision tree may include multiple decision trees of the same or different depths, and when training a gradient boosting decision tree, the difference between the prediction result of the previous tree for a sample and the true label of the sample is used as the label of the sample when training the next tree.

[0049] Figure 2 This diagram illustrates the generation of features using a gradient boosting decision tree according to an exemplary embodiment. (Refer to...) Figure 2 For ease of description, it is assumed here that the gradient boosting decision tree consists of two trees of depth 2, and the samples used to train the gradient boosting decision tree are s1, s2, s3, s4, and s5. If, during the training of the first tree (tree 1), samples s1, s2, and s3 fall on the leaf node numbered 2 in tree 1, and samples s4 and s5 fall on the leaf node numbered 3 in tree 2, and during the training of the second tree (tree 2), samples s2 and s5 fall on the leaf node numbered 2 in tree 2, and samples s1, s3, and s4 fall on the leaf node numbered 3 in tree 2, then... Figure 2 As shown in the table, the leaf node number 2 of tree 1 and the leaf node number 3 of tree 2 where sample s1 falls are used as features related to the generated data record corresponding to sample s1. Similarly, the leaf node number 2 of tree 1 and the leaf node number 3 of tree 2 where sample s3 falls are used as features related to the generated data record corresponding to sample s3. It should be noted that the leaf node numbering method can be arbitrary and is not limited to this. Figure 2 The numbering method in [the document / document].

[0050] Alternatively, in response to the presence of temporal features in the dataset, the feature engineering strategy determination device 120 can determine the feature engineering strategy for the dataset as follows: sort the multiple data records according to temporal features; during the process of training a logistic regression model sequentially using the sorted data records, obtain the temporal features corresponding to the current data record based on the parameter terms related to the model parameters of the logistic regression model, and use these as the generated features related to each data record. Here, the presence of temporal features in the dataset can be a meta-feature reflecting the overall characteristics of the dataset. Specifically, for example, after sorting the multiple data records according to temporal features, the logistic regression model can be trained first using at least one data record that is first sorted according to temporal features. Then, the parameter terms related to the model parameters of the trained logistic regression model can be used as the temporal features corresponding to the current data record immediately following the at least one data record. Subsequently, the logistic regression model can be updated using the data record immediately following the at least one data record, and the parameter terms related to the model parameters of the currently updated logistic regression model can be used as the temporal features corresponding to the next data record following that data record. In other words, the logistic regression model is continuously updated, and the parameter terms related to the model parameters of the logistic regression model updated using data records prior to the current data record are used as time-series features corresponding to the current data record. As an example, parameter terms related to the model parameters may include the model parameters themselves, the results of calculations between model parameters, and / or statistical information of the model parameters during the model update process. For example, the results of calculations between model parameters may be the results of performing predetermined operations (e.g., summation, averaging, etc.) between multiple model parameters of the same logistic regression model. Besides using the updated logistic regression model parameters and / or the results of calculations between model parameters as time-series features, optionally, since the model parameters change continuously during the logistic regression model update process, the statistical information of the model parameters during the model update process can also be used as time-series features. Specifically, the statistical information may be, for example, the mean, variance, standard deviation, etc., of at least one model parameter during the model update process, but is not limited to these. It should be noted that although the feature engineering strategy here is defined as obtaining time series features by using the parameters of the logistic regression model, this application is not limited to this. The logistic regression model can be replaced with other machine learning models, or the time series features corresponding to the current data record can be obtained by statistical analysis of the data records before the current data record after sorting by time features.

[0051] Alternatively, in response to the inclusion of multiple time features in the dataset, the feature engineering strategy determination device 120 can determine the feature engineering strategy for the dataset as follows: for each data record, generate features related to each data record by subtracting the pairwise combinations of the multiple time features. For example, assuming that the data record includes three time features t1, t2, and t3, then t1-t2, t2-t3, and t3-t1 can be used as the generated features related to each data record.

[0052] It should be noted that the above four feature engineering strategies can be combined. For example, if the obtained meta-features indicate both the presence of temporal features in the dataset and that continuous features are stronger than discrete features in the distribution, then the feature engineering strategies determined in response to the presence of temporal features in the dataset and the stronger distribution of continuous features than discrete features can be combined and executed. Furthermore, in addition to the above-mentioned method of determining feature engineering strategies based on meta-features, the feature engineering strategy determination device 120 can further combine other feature engineering methods, such as performing principal component analysis (PCA), singular value decomposition (SVD), and neural network stacking on the data records in the dataset.

[0053] After the feature engineering strategy determination device 120 determines the feature engineering strategy, the feature generation device 130 can use the determined feature engineering strategy to generate features related to each data record in the dataset, that is, generate features related to each data record in the dataset according to the feature generation method indicated in the determined feature engineering strategy.

[0054] As mentioned above, after obtaining the dataset, this application can automatically determine the feature engineering strategy for the dataset through meta-learning, thereby eliminating the reliance on the personal experience and familiarity with business scenarios of machine learning experts and relevant business personnel when determining the feature engineering strategy during the training of machine learning models. Therefore, it not only improves the automation level of the entire model training and lowers the threshold of machine learning, but also enables the feature engineering strategy for the dataset to be determined more objectively through quantitative analysis. As a result, the features generated by using the objectively determined feature engineering strategy are more conducive to the subsequent training of machine learning models.

[0055] The sample construction apparatus 140 can construct machine learning samples based at least on the generated features. As an example, the sample construction apparatus 140 can construct machine learning samples based solely on the generated features. Alternatively, the sample construction apparatus 140 can also construct machine learning samples based on both the generated features and the original features included in the data record, for example, by merging the generated features with the original features included in the data record.

[0056] The training device 150 can train a machine learning model using machine learning samples constructed by the sample construction device 140 based on automatically obtained hyperparameter values. For example, firstly, the training device 150 can use a portion of all constructed machine learning samples as a training sample set. Here, another portion of all machine learning samples can be used as a validation sample set to verify the model performance of the trained machine learning model. Secondly, the training device 150 can automatically generate multiple sets of hyperparameter values ​​that affect the training of the machine learning model using different hyperparameter generation strategies, depending on the number of training samples included in the training sample set. Subsequently, the training device 150 can train multiple machine learning models using the constructed machine learning samples based on each of the multiple sets of hyperparameter values. Finally, the training device 150 can select several machine learning models with the highest model performance from the multiple trained machine learning models and fuse the selected machine learning models using a predetermined model fusion strategy to determine the final trained machine learning model.

[0057] Next, we will first describe in detail the process by which the training device 150 automatically adopts different hyperparameter generation strategies to generate multiple sets of hyperparameter values ​​that affect the training of the machine learning model, based on the number of training samples included in the training sample set.

[0058] Generally, the smaller the number of training samples in a dataset, the smaller the range of hyperparameter values. Finding suitable hyperparameters within a smaller range requires less computational resources and is more likely to be found quickly. Therefore, in this case, hyperparameter values ​​can be generated randomly. Specifically, in response to the number of training samples in the training sample set being less than or equal to a preset threshold, the training device 150 can randomly generate multiple sets of hyperparameter values ​​within the range corresponding to the number of training samples; alternatively, it can predefine the hyperparameter values ​​for some hyperparameters and, for the remaining hyperparameters, randomly generate multiple sets of values ​​within the range corresponding to the number of training samples. For example, if the number of training samples is less than the preset threshold, some hyperparameters with a smaller impact on the model performance can be fixed based on prior experience, while hyperparameters with a larger impact on the model performance can have multiple sets of values ​​randomly generated within the range corresponding to the number of training samples.

[0059] However, the larger the number of training samples, the wider the range of hyperparameter values ​​becomes when generating hyperparameters. Consequently, the wider the range of hyperparameter values, the more difficult it is to find suitable hyperparameter values, and the more computational resources are consumed in finding suitable hyperparameter values ​​(resulting in lower efficiency). Therefore, this application also proposes a concept for generating hyperparameter values ​​that can be used in the training process of any machine learning model.

[0060] Specifically, in response to the number of training samples in the training sample set exceeding the preset threshold, the training device 150 can generate multiple sets of hyperparameter values ​​in the following manner: predefine hyperparameter values ​​for some hyperparameters, and for the remaining hyperparameters, generate multiple sets of hyperparameter values ​​using a combination of randomly generating hyperparameter values ​​within a range corresponding to the number of training samples and generating hyperparameter values ​​using a Bayesian optimization model. Here, the Bayesian optimization model can be constructed to reflect the relationship between hyperparameter values ​​and their corresponding model performance. It should be noted that although some hyperparameter values ​​are predefined here, multiple sets of hyperparameter values ​​can also be generated for all hyperparameters using the above combination method.

[0061] Next, the method of generating multiple sets of hyperparameter values ​​by combining other hyperparameters in this application will be described in detail.

[0062] First, the training device 150 can construct a sub-training sample set by sampling the training samples in the training sample set. Here, the training device 150 is not limited to constructing a sub-training sample set by sampling the training samples in the training sample set only once, but can also construct multiple sub-training sample sets with different numbers of training samples by sampling the training samples in the training sample set multiple times. Furthermore, in the multiple sampling, the sub-training sample set obtained by the later sampling includes more training samples than the previous sampling.

[0063] According to an exemplary embodiment, the training device 150 can construct a sub-training sample set by performing density-based subsampling on the training samples in the training sample set. Here, density-based subsampling can be sampling based on the distribution density of the training samples in a specific space, which is more likely than random sampling to reflect the distribution pattern of the samples in the entire training sample set. For example, the training device 150 can reduce the dimensionality of the training samples in the training sample set to another dimensional space, and construct the sub-training sample set by sampling the training samples according to the labels of the training samples and the distribution density of the training samples in the training sample set in the other dimensional space. Here, the other dimensional space is a dimensional space with a lower dimension than the training samples themselves, such as a two-dimensional plane or a three-dimensional space. For example, assuming that the training samples include 100 features (i.e., each training sample corresponds to a 100-dimensional vector), it is usually difficult to sample based on the distribution density of the training samples in such a high-dimensional space, but if it is reduced to a lower-dimensional space, density-based subsampling can be facilitated.

[0064] For ease of description, the subsampling process according to this application will be described below using the example of reducing the training samples in the training sample set to a two-dimensional plane.

[0065] First, the training device 150 maps each training sample in the training sample set to a sample point on a two-dimensional plane. Specifically, the training device 150 randomly generates a corresponding two-dimensional vector for each continuous feature in each training sample, and randomly generates a corresponding two-dimensional vector for each value of each discrete feature in each training sample. Then, the training device 150 performs a weighted sum of the two-dimensional vectors corresponding to each feature in each training sample to obtain the coordinates of the training sample on the two-dimensional plane. Regarding the weights of each feature, for continuous features, the weight of the corresponding two-dimensional vector can be the value of the continuous feature; for discrete features, if the discrete feature has a value in the training sample, the weight of the discrete feature can be set to 1, and if the discrete feature has no value in the training sample, the weight of the discrete feature can be set to 0.

[0066] Secondly, the training device 150 can classify all sample points on the two-dimensional plane into positive sample point sets and negative sample point sets based on whether the training samples are positive or negative samples. Then, for each point set in the positive and negative sample point sets, the sample points can be sampled according to a set sampling ratio based on the distribution density of the sample points in that set, and the set of training samples corresponding to the sampled sample points is taken as a sub-training sample set. For example, for each point set, the training device 150 can determine a planar region (e.g., a rectangular region, a square region, etc., but not limited to) covering all sample points in the point set based on the maximum and minimum values ​​of the x-coordinate and y-coordinate of all sample points in that point set on the two-dimensional plane, and divide the determined planar region evenly into multiple sub-regions (e.g., dividing the length and width of a rectangular region into ten parts each, thus dividing it into 100 rectangular sub-regions) and count the number of sub-regions into which sample points fall. In response to the number of sub-regions into which sample points fall being greater than a preset number (e.g., 25), the training device 150 can randomly sample in each sub-region according to a set sampling ratio. However, in response to the number of sub-regions in which sample points fall being less than or equal to the preset number, the training device 150 may increase the number of sub-regions until the number of sub-regions in which sample points fall being greater than the preset number, and then randomly sample in each sub-region according to a set sampling ratio (for example, randomly sampling 10% of all sample points in each sub-region). Here, increasing the number of sub-regions may involve further subdividing the sub-regions in which sample points fall. Furthermore, after obtaining sample points through sampling on a two-dimensional plane, the training sample corresponding to the sample point can be determined through inverse mapping.

[0067] The sampling method described above according to the exemplary embodiments of this application not only ensures the ratio of positive to negative samples in the training sample set, but also ensures that the distribution of training samples in the sub-training sample set is substantially the same as the distribution of training samples in the training sample set. Therefore, subsequently, using such a sub-training sample set to generate hyperparameter values ​​not only ensures that the generated hyperparameter values ​​have a certain degree of fidelity, but also reduces the time and computational resources required to generate hyperparameter values, thereby improving the efficiency of model training.

[0068] After constructing the sub-training sample set, for the remaining hyperparameters, the training device 150 can randomly generate predetermined sets of initial hyperparameter values ​​within a range corresponding to the number of training samples included in the sub-training sample set. Based on each set of hyperparameter values, which consists of predefined values ​​of some hyperparameters and each set of initial hyperparameter values ​​for the remaining hyperparameters, the machine learning model is trained using the sub-training sample set, and the model performance of the trained machine learning model is validated to obtain the model performance corresponding to each set of hyperparameter values. Here, the number of predetermined sets of initial hyperparameter values ​​depends on the number of hyperparameter values ​​and their corresponding model performances required to build the Bayesian optimization model. Typically, the number of hyperparameter values ​​and their corresponding model performances required to build different Bayesian optimization models is different. Furthermore, when validating the model performance, all the machine learning samples constructed above, excluding the validation samples used as the training sample set, can be used. The model performance can be obtained by comparing the prediction results of the trained machine learning model for each validation sample in the validation sample set with the true label of that validation sample.

[0069] Subsequently, the training device 150 can establish a Bayesian optimization model based on each set of hyperparameter values ​​and their corresponding model effects. Establishing a Bayesian optimization model can be a process of continuously establishing the relationship between hyperparameter values ​​and their corresponding model effects. Once the Bayesian optimization model is established, hyperparameter values ​​corresponding to the desired model effect can be generated when needed based on the correspondence between hyperparameter values ​​and model effects.

[0070] Subsequently, the training device 150 can continuously determine the number of hyperparameter values ​​that still need to be generated for the remaining hyperparameters, generate a portion of this number using a Bayesian optimization model, and randomly generate another portion of this number within the range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set. For example, assuming that 10 machine learning models can be trained in parallel each time due to system resource limitations, after the Bayesian optimization model is established, the training device 150 can first generate 10 sets of hyperparameter values ​​in a combination manner, and train 10 machine learning models based on each set of hyperparameter values ​​using the sub-training sample set. If it is determined that 4 machine learning models have been trained, the training device determines that the number of hyperparameter values ​​that still need to be generated for the remaining hyperparameters is 4 (that is, 4 sets of hyperparameter values ​​that need to be generated for the remaining hyperparameters). Then, it can generate 2 sets of hyperparameter values ​​using the Bayesian optimization model, and randomly generate 2 sets of hyperparameter values ​​within the range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set. Although in the above example, the hyperparameter values ​​to be generated are allocated to the random generation method and the generation method using the Bayesian optimization model in a 1:1 ratio, this application does not limit the ratio of one part of the above set of numbers to the other part of the above set of numbers. For example, the ratio can be 1:1 or 1:2 or any other ratio.

[0071] Additionally, it should be noted that after the Bayesian optimization model is established, whenever new hyperparameter values ​​and their corresponding model effects are obtained, the Bayesian optimization model is updated. This allows the Bayesian optimization model to be continuously optimized to generate hyperparameter values ​​that are more likely to be suitable for model training. The updated Bayesian optimization model can then generate new hyperparameter values ​​in proportion to an allocated set, along with a random generation method.

[0072] Furthermore, it should be noted that in this application, the training device 150 can execute in parallel the operations of generating a portion of the set of numbers using the Bayesian optimization model and the operations of randomly generating another portion of the set of numbers. Specifically, the operations of generating a portion of the set of numbers using the Bayesian optimization model and the operations of randomly generating another portion of the set of numbers can be assigned to different processing processes (threads), thereby helping to shorten the overall training time of the model. In addition, compared with the serial Bayesian optimization model generation method or the random generation method, the hyperparameter value generation method that executes the random generation method and the generation method using the Bayesian optimization model in parallel not only reduces time resources, but also makes it more conducive to generating hyperparameter values ​​that are truly suitable for model training with less computational resource consumption. This is because simply generating hyperparameters randomly is inherently random, while Bayesian optimization models are predictive (in other words, Bayesian optimization models can provide the directionality of hyperparameter values ​​based on the relationship between hyperparameter values ​​and model performance, but the accuracy of this directionality depends on the accuracy of the Bayesian optimization model). If only random generation is used to generate hyperparameter values, the randomness may require many attempts to find hyperparameter values ​​that result in better model performance. Conversely, if only Bayesian optimization models are used, it is difficult to find truly suitable hyperparameter values ​​in a short time when the predicted direction deviates. The combined approach of randomly generating hyperparameter values ​​and using Bayesian optimization models proposed in this application not only reduces the time and computational resources consumed in model training but also ensures the generation of suitable hyperparameter values.

[0073] As described above, multiple sub-training sample sets with an ever-increasing number of training samples can be obtained through multiple sampling. In this case, whenever new hyperparameter values ​​are generated for the remaining hyperparameters according to the aforementioned combination method, the training device 150 can train the machine learning model using the sub-training sample set obtained from the first sampling, based on each set of hyperparameter values ​​consisting of the predefined hyperparameter values ​​of the partial hyperparameters and the generated new hyperparameter values. Then, the training device 150 selects the multiple sets of hyperparameter values ​​with the highest corresponding model performance from all sets of hyperparameter values ​​used when training the machine learning model using the sub-training sample set obtained from the previous sampling, and uses these as all sets of hyperparameter values ​​to be used when training the machine learning model using the sub-training sample set obtained from the next sampling. The training device 150 can continue to perform the above operations until a predetermined condition is met, at which point sampling stops, and the multiple sets of hyperparameter values ​​with the highest corresponding model performance from all sets of hyperparameter values ​​used when training the machine learning model using the sub-training sample set obtained from the last sampling are used as multiple sets of hyperparameter values ​​that influence the training of the machine learning model.

[0074] For example, after the training device 150 obtains the first sub-training sample set through the first sampling, whenever a new hyperparameter value is generated, a machine learning model can be trained based on each set of hyperparameter values ​​consisting of the predefined hyperparameter values ​​of the partial hyperparameters and the generated new hyperparameter values. The model effect corresponding to each set of hyperparameter values ​​is obtained by validating the machine learning model using a validation sample set. For example, if the number of sets of hyperparameter values ​​for which the corresponding model effect has been obtained exceeds a preset threshold, for example, 40 sets, then the top-ranking sets of hyperparameter values ​​for the corresponding model effect can be selected from these 40 sets (for example, 20 sets of hyperparameter values). Then, based on each of the selected 20 sets of hyperparameter values, the machine learning model is trained using the second sub-training sample set obtained from the second sampling (the second sub-training sample set includes more training samples than the first sub-training sample set). The model effect corresponding to each set of hyperparameter values ​​is obtained by validating the machine learning model using a validation sample set, and the top-ranking sets of hyperparameter values ​​for the corresponding model effect are selected from these sets (for example, 10 sets of hyperparameter values). Subsequently, the training device 150 can continue to train machine learning models (i.e., train 10 machine learning models) based on each of the selected 10 sets of hyperparameter values ​​using a third sub-training sample set obtained from the third sampling (the third sub-training sample set includes more training samples than the second sub-training sample set). It then obtains the model performance corresponding to each set of hyperparameter values ​​by validating the machine learning models using a validation sample set. From these 10 sets of hyperparameter values, it selects several sets of hyperparameter values ​​with the highest corresponding model performance (e.g., 3 sets of hyperparameter values) as multiple sets of hyperparameter values ​​influencing the training of the machine learning models. For ease of description, it is assumed here that the sampling is three times, but it should be noted that this application does not limit the number of samplings or the number of sets of hyperparameter values ​​with the highest model performance selected each time. Furthermore, several sets of hyperparameter values ​​can be recorded in the storage device (not shown) of the automatic model training system for later use.

[0075] It should be noted that the hyperparameters mentioned in this application can be any hyperparameters that may affect the training of a machine learning model, and can vary depending on the machine learning model. For example, for a neural network model, hyperparameters can be the number of hidden layers, the number of nodes in each hidden layer, the learning rate, etc. Furthermore, although the description of hyperparameter value generation above mentions that some hyperparameter values ​​can be predefined, and multiple sets of hyperparameter values ​​can be generated for other hyperparameters through combinations, alternatively, some hyperparameter values ​​may not be predefined. Instead, multiple sets of hyperparameter values ​​can be generated for all hyperparameters through combinations. Alternatively, multiple sets of hyperparameter values ​​can be automatically generated for different types of hyperparameters using different methods. For example, for architecture-related hyperparameters (e.g., the number of hidden layers, the number of nodes in each hidden layer, the activation function of each hidden layer, etc.), multiple sets of hyperparameter values ​​can be generated using neural architecture search. For training-related hyperparameters (e.g., learning rate, embedding dimension, etc.), multiple sets of hyperparameter values ​​can be generated using the combination methods or random generation methods described above.

[0076] As described above, after generating multiple sets of hyperparameter values ​​that influence the training of the machine learning model, the training device 150 can train multiple machine learning models using the constructed machine learning samples based on each set of hyperparameter values. Here, the constructed machine learning samples can be a portion of the training samples in the training sample set, or they can be all the training samples in the training sample set. For example, if the number of training samples in the training sample set is less than a preset threshold, the entire training sample set can be used directly to train the machine learning model; if the number of training samples in the training sample set is greater than the preset threshold, a portion of the training samples in the training sample set can be used to train the machine learning model.

[0077] After training multiple machine learning models, the training device 150 can select the top-performing models from the trained models and fuse them using a predetermined model fusion strategy to determine the final trained machine learning model. Specifically, when the number of training samples in the training sample set exceeds the preset threshold, since the machine learning model was previously trained using a sub-training sample set obtained by sampling the training samples instead of the entire training sample set, the training sample training device 150 can retrieve from a storage device (not shown) several sets of hyperparameter values ​​used when training the top-performing machine learning models (the machine learning models trained using the sub-training sample set). Based on each set of hyperparameter values, the machine learning model is retrained using the entire training sample set, and the retrained machine learning models are fused using a predetermined model fusion strategy to determine the final trained machine learning model, thereby further improving the model training effect. Furthermore, in this application, the automatic model training system 100 can store the number of training rounds in which the machine learning model is trained twice using the entire training sample set based on each set of hyperparameter values ​​in a storage device (not shown) for later use.

[0078] The process of model fusion according to this application will be briefly described below.

[0079] The training device 150 can first use the plurality of machine learning models as an initial model candidate set. Then, it performs the following model combination operation on the initial model candidate set: combining the machine learning model with the best performance among the plurality of machine learning models with the machine learning models in the initial model candidate set, and calculating the model performance of each combined machine learning model group; determining whether the model performance of each combined machine learning model group is improved compared to the best model performance before combination; if the model performance of the combined machine learning model group is not improved compared to the best model performance before combination, determining the machine learning model with the best performance before combination as the final machine learning model; if the model performance of the combined machine learning model group is improved compared to the best model performance before combination, adding the combined machine learning model group with the best performance after combination to the initial model selection set to form the current model candidate set. Next, the training device 150 can repeat the same model combination operation as the one performed on the initial model candidate set for the current model candidate set a predetermined number of times, and select the machine learning model combination with the best performance from the final generated current model candidate set. Finally, based on the number of each machine learning model included in the machine learning model combination, the weights of the machine learning models during model fusion are determined.

[0080] The model fusion process of this application is briefly described below with an example. Assume that the three machine learning models M1, M2, and M3 with the highest performance are selected, and their corresponding model performances are 0.7, 0.8, and 0.9, respectively. The initial model candidate set is {M1, M2, M3}. Since M3 has the best performance in the initial candidate set, M3 can be combined with M1, M2, and M3 from the initial candidate set to obtain combined machine learning model groups [M1, M3], [M2, M3], and [M3, M3], and the model performance of the combined machine learning models is calculated. As an example, the model performance of each combined machine learning model group can be calculated by comparing the average prediction result of each machine learning model in the combined machine learning model group for the validation sample set with the true labeled value of the validation sample set. Suppose the calculated model performance of the machine learning model group [M1,M3], [M2,M3], and [M3,M3] is 0.85, 0.92, and 0.9, respectively. Since the model performance of [M2,M3] is greater than that of the optimal machine learning model M3 before combination, the machine learning model group [M2,M3] can be added to the current model candidate set. The model candidate set at this time is {M1,M2,M3, [M2,M3]}. Then, the best-performing model [M2,M3] in the current model candidate set is combined with M1,M2,M3 and [M2,M3] in the current model candidate set, respectively. The model performance of the combined machine learning model groups [M1,M2,M3], [M2,M2,M3], [M3,M2,M3] and [M2,M3,M2,M3] is calculated. If the model performance of [M1,M2,M3] is better than that of [M2,M3], then [M1,M2,M3] is added to the model candidate set. The current model candidate set is then {M1,M2,M3, [M2,M3], [M1,M2,M3]}. Suppose that after multiple iterations, the optimal combination of machine learning models in the final candidate model set is [M1, M2, M2, M3, M3]. Since there is 1 M1, 2 M2, and 2 M3 in this combination, the weights of machine learning models M1, M2, and M3 during model fusion can be determined as 1 / 5, 2 / 5, and 2 / 5, respectively.

[0081] At this point, the training device 150 has obtained several sets of hyperparameter values ​​that have the best model performance on the validation sample set (in the example above, for example, the three sets of hyperparameter values ​​that have the best model performance are obtained, and three machine learning models trained based on each of the three sets of hyperparameter values ​​are obtained respectively), the number of rounds of training the machine learning model using the training sample set based on each set of hyperparameter values, and the weights of each machine learning model when the models are fused (e.g., 1 / 5, 2 / 5, and 2 / 5), and these can be stored in the storage device.

[0082] However, since the machine learning model trained above is the best performing model on the validation sample set, and the validation sample set itself did not participate in the model training, optionally, according to the exemplary embodiment of this application, the training device 150 may also obtain several sets of hyperparameter values ​​used when training the top-performing machine learning models, and the number of rounds of training the machine learning model using the training sample set based on each of the several sets of hyperparameter values. Based on each set of hyperparameter values, the training device 150 may then train the machine learning model for the corresponding number of rounds using all constructed machine learning samples (all machine learning samples after the merger of the training and validation sample sets) to obtain several retrained machine learning models. Finally, the training device 150 may fuse the several retrained machine learning models according to their previously determined weights during model fusion to obtain the final trained machine learning model, which is the best performing machine learning model on all constructed machine learning samples.

[0083] Although the above uses a weighted sum method for model fusion, the fusion method is not limited to this. It can also be an average fusion method, a voting fusion method, etc. Furthermore, in this application, different types of machine learning models can be fused, not just the same type of machine learning models trained under different hyperparameter values.

[0084] Furthermore, the machine learning model mentioned in this application can be a neural network model (e.g., a deep sparse network) or a logistic regression model, but is not limited to these; rather, it can be any machine learning model.

[0085] The above has been referred to Figure 1 An automatic model training system according to an exemplary embodiment of this application is described, which can automatically determine a feature engineering strategy through meta-learning, generate sample features using the determined feature engineering strategy to construct machine learning samples, and train a machine learning model using the constructed machine learning samples with automatically obtained hyperparameters, thereby greatly reducing the dependence on machine learning experts in model training and improving model training efficiency while ensuring training effect.

[0086] It should be noted that although the above description of the automatic model training system divides it into devices for performing corresponding processes (e.g., dataset acquisition device 110, feature engineering strategy determination device 120, feature generation device 130, sample construction device 140, and training device 150), those skilled in the art will understand that the processes performed by the above devices can also be performed without any specific device division or clear boundaries between the devices in the automatic model training system. Furthermore, the above references... Figure 1 The described automatic model training system 100 is not limited to the devices described above, but may also include other devices (e.g., storage devices, data processing devices, etc.) as needed, or the above devices may be combined.

[0087] It should be noted that the implementation of exemplary embodiments of the present invention does not rely on any limitations on specific business logic, but focuses on a relatively general scenario. Unlike most existing solutions, exemplary embodiments of the present invention are not focused on a specific scenario, but can be applied to a variety of different scenarios, such as recommendation systems, advertising systems, banking systems, etc. Furthermore, based on exemplary embodiments of the present invention, modelers can continue to incorporate their own business experience to further improve the model training effect. Therefore, exemplary embodiments of the present invention consider the abstraction of application scenarios, are not targeted at any specific scenario, but are applicable to any scenario.

[0088] In other words, according to exemplary embodiments of the present invention, the data in the dataset can be image data, voice data, text data, business data, such as data describing engineering control objects, data describing users (or their behavior), data describing objects and / or events in various fields such as administration, commerce, medicine, regulation, and finance. Accordingly, the machine learning model mentioned in this application aims to make predictions about problems related to the above-mentioned objects or events in order to help users make any decisions. For example, the model can be used to predict image categories, text categories, voice sentiment, fraudulent transactions, advertising click-through rates, etc., so that the prediction results can be directly used as a basis for decision-making or further combined with other rules to become a basis for decision-making. Exemplary embodiments of the present invention do not limit the specific technical fields involved in the prediction purpose of the model, but the reason is that the model is fully applicable to any specific field or scenario that can provide corresponding training data or prediction data, and does not mean that the model cannot be applied to the relevant technical fields.

[0089] In other words, after obtaining the automatically trained machine learning model as described above, further predictive data records can be collected and input into the machine learning model to obtain corresponding prediction results. These prediction results can then be used to make appropriate decisions. For example, financial business data can be collected, and features related to the financial business data can be generated according to feature engineering strategies. At least based on the generated features, prediction samples can be constructed, and the prediction results provided by the machine learning model for the prediction samples can be obtained. Based on the prediction results, activities such as financial risk control, marketing, and debt collection can be carried out. For example, warnings or notifications corresponding to financial risk control, marketing, and debt collection can be generated and output to decision-making entities or relevant users. The aforementioned financial business data can also be internet business-related data, production control data, etc.

[0090] Figure 3 A flowchart is shown for a method for automatically training a machine learning model according to an exemplary embodiment (hereinafter, for ease of description, it is simply referred to as the "automatic model training method").

[0091] Here, as an example, Figure 3 The automatic training method for the model shown can be derived from... Figure 1 The model automatic training system 100 shown can be executed, but it can also be implemented entirely in software via computer programs or instructions. Furthermore, it can be executed via a specifically configured computing system or computing device, for example, via a system including at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to execute the aforementioned machine learning method. For ease of description, it is assumed that... Figure 3 The method shown is by Figure 1 The model automatic training system 100 shown is used to perform this, and it is assumed that the model automatic training system 100 can have Figure 1 The configuration shown.

[0092] Reference Figure 3 In step S310, the dataset acquisition device 110 can acquire a dataset including multiple data records. Here, the dataset may involve image data, voice data, text data, and / or business data, but is not limited to these.

[0093] In step S320, the feature engineering strategy determination device 120 can determine a feature engineering strategy for the dataset through meta-learning. For example, in step S320, the feature engineering strategy determination device 120 can analyze the dataset to obtain meta-features reflecting the overall characteristics of the dataset, and determine a feature engineering strategy for the dataset based on the obtained meta-features. For example, the feature engineering strategy determination device 120 can analyze the dataset to obtain statistical information related to the number of data records included in the dataset, the types of features included in the data records, the number of features, and / or the values ​​of the features, as meta-features. As an example, meta-features may include at least one of the following: the number of data records included in the dataset, the number of discrete features included in the data records, the number of continuous features included in the data records, whether there are time features in the data records, the number of time features included in the data records, the fill factor of each discrete feature, the fill factor of each continuous feature, the average fill factor of all discrete features and the average fill factor of all continuous features, the mean, variance, median, maximum or minimum value of each continuous feature, or the number of different values ​​of each discrete feature, or the result obtained by combining and statistically analyzing the above items, but is not limited thereto.

[0094] After obtaining the meta-features of the dataset, for example, the feature engineering strategy determination device 120 can automatically determine a feature engineering strategy for the dataset based on the correspondence between the meta-features and feature engineering strategies. Optionally, the feature engineering strategy determination device 120 can also determine a feature engineering strategy for the dataset using another machine learning model based on the obtained meta-features. As an example, the feature engineering strategy determination device 120 can use a clustering model, which is the other machine learning model, to determine historical datasets similar to the dataset that already have corresponding model training results, and determine the feature engineering strategy corresponding to the historical dataset as the feature engineering strategy for the dataset. As another example, the feature engineering strategy determination device 120 can use a prediction sample input based on the obtained meta-features as the prediction model of the other machine learning model, and use the prediction result of the prediction model for the prediction sample as the feature engineering strategy for the dataset. Here, the prediction model can be pre-trained to predict the feature engineering strategy corresponding to the meta-features based on the meta-features.

[0095] As an example, when determining a feature engineering strategy for a dataset based on the obtained meta-features, the feature engineering strategy determination device 120 may perform at least one of the following operations: in response to the fact that discrete features are stronger than continuous features in distribution, the feature engineering strategy for the dataset is determined as follows: discretizing continuous features to obtain discretized features, and searching for a combination of discrete features beneficial to model training effect in the Cartesian product combination of the original discrete features and the discretized features, as the generated features related to each data record; in response to the fact that continuous features are stronger than discrete features in distribution, the feature engineering strategy for the dataset is determined as follows: training a gradient boosting decision tree model using the dataset, and determining the features based on the samples formed by each data record. The leaf node number of each decision tree is used as the generated feature related to each data record; in response to the presence of time features in the dataset, the feature engineering strategy for the dataset is determined as follows: the multiple data records are sorted according to the time features, and during the process of training the logistic regression model sequentially using the sorted data records, the time-series features corresponding to the current data record are obtained based on the parameter terms related to the model parameters of the logistic regression model, and used as the generated feature related to each data record; in response to the inclusion of multiple time features in the dataset, the feature engineering strategy for the dataset is determined as follows: for each data record, the feature related to each data record is generated by subtracting the multiple time features in pairs.

[0096] In step S330, the feature generation device 130 can generate features related to each data record in the dataset using the feature engineering strategy determined in step S320. Next, in step S340, the sample construction device 140 can construct machine learning samples based at least on the generated features.

[0097] Finally, in step S350, the training device 150 can train a machine learning model using the constructed machine learning samples based on the automatically obtained hyperparameter values.

[0098] For example, in step S350, the training device 150 can use a portion of all the constructed machine learning samples as a training sample set, and automatically use different hyperparameter generation strategies to generate multiple sets of hyperparameter values ​​that affect the training of the machine learning model according to the number of training samples included in the training sample set. Based on each set of hyperparameter values, multiple machine learning models are trained using the constructed machine learning samples. Then, the top-ranked machine learning models are selected from the trained machine learning models, and the selected machine learning models are fused using a predetermined model fusion strategy to determine the final trained machine learning model.

[0099] Specifically, in response to the number of training samples in the training sample set being less than or equal to a preset threshold, the training device 150 can generate multiple sets of hyperparameter values ​​in the following ways: randomly generating multiple sets of hyperparameter values ​​within a range of hyperparameter values ​​corresponding to the number of training samples; or, predefining hyperparameter values ​​for some hyperparameters and, for the remaining hyperparameters, randomly generating multiple sets of hyperparameter values ​​within a range of hyperparameter values ​​corresponding to the number of training samples. In response to the number of training samples in the training sample set being greater than the preset threshold, the training device 150 can generate multiple sets of hyperparameter values ​​in the following ways: predefining hyperparameter values ​​for some hyperparameters and, for the remaining hyperparameters, generating multiple sets of hyperparameter values ​​using a combination of randomly generating hyperparameter values ​​within a range of hyperparameter values ​​corresponding to the number of training samples and generating hyperparameter values ​​using a Bayesian optimization model. Here, the Bayesian optimization model can be constructed to reflect the relationship between hyperparameter values ​​and their corresponding model performance. Furthermore, after the Bayesian optimization model is established, whenever new hyperparameter values ​​and their corresponding model effects are obtained, the training device 150 can use them to update the Bayesian optimization model.

[0100] Specifically, when generating hyperparameter values ​​for other hyperparameters according to a combination method, a sub-training sample set can first be constructed by sampling the training samples in the training sample set. For example, the training device 150 can construct the sub-training sample set by performing density-based subsampling on the training samples in the training sample set. Specifically, the training device 150 can reduce the training samples in the training sample set to another dimension space, and construct the sub-training sample set by sampling the training samples according to the label of the training samples and the distribution density of the training samples in the training sample set in the other dimension space. For example, the training device 150 can map each training sample in the training sample set to a sample point on a two-dimensional plane. Subsequently, the training device 150 can classify all sample points on the two-dimensional plane into a positive sample point set and a negative sample point set according to whether the training sample is a positive sample or a negative sample. For each point set in the positive sample point set and the negative sample point set, the sample points are sampled according to the distribution density of the sample points in the point set according to a set sampling ratio, and the set consisting of the training samples corresponding to the sampled sample points is taken as the sub-training sample set. Specifically, for example, the training device 150 can determine a planar region covering all sample points in the point set based on the maximum and minimum values ​​of the x and y coordinates of all sample points in the point set on a two-dimensional plane. The determined planar region is then divided into multiple sub-regions on an even scale, and the number of sub-regions into which sample points fall is counted. Subsequently, in response to a sub-region with a sample point falling into it exceeding a preset number, the training device 150 can perform random sampling in each sub-region according to a set sampling ratio. In response to a sub-region with a sample point falling into it being less than or equal to the preset number, the training device 150 can increase the number of sub-regions until the number of sub-regions with a sample point falling into it exceeds the preset number, and then perform random sampling in each sub-region according to a set sampling ratio.

[0101] Subsequently, for the remaining hyperparameters, predetermined sets of initial hyperparameter values ​​are randomly generated within a range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set. Based on each set of hyperparameter values, which consists of predefined values ​​of some hyperparameters and each set of initial hyperparameter values ​​for the remaining hyperparameters, the machine learning model is trained using the sub-training sample set. The model performance of the trained machine learning model is then validated to obtain the model performance corresponding to each set of hyperparameter values. Next, a Bayesian optimization model is established based on each set of hyperparameter values ​​and its corresponding model performance. After the Bayesian optimization model is established, the training device 150 can continuously determine the number of sets of hyperparameter values ​​that still need to be generated for the remaining hyperparameters, generate a portion of this number using the Bayesian optimization model, and randomly generate another portion of this number within a range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set. Here, the operation of generating a portion of this number using the Bayesian optimization model and the operation of randomly generating the other portion of this number can be performed in parallel.

[0102] It should be noted that sampling training samples from the training sample set to construct a sub-training sample set can involve multiple samplings of the training samples in the training sample set to construct multiple sub-training sample sets with different numbers of training samples. Furthermore, in each of these multiple samplings, the sub-training sample set obtained from a later sampling may include more training samples than the sub-training sample set obtained from a previous sampling. In this case, whenever new hyperparameter values ​​are generated according to the aforementioned combination method, the training device 150 can train the machine learning model using the sub-training sample set obtained from the first sampling, based on each set of hyperparameter values ​​consisting of the predefined hyperparameter values ​​of the aforementioned partial hyperparameters and the generated new hyperparameter values. Subsequently, among all the sets of hyperparameter values ​​used when training the machine learning model using the sub-training sample set obtained from the previous sampling, the sets of hyperparameter values ​​with the highest corresponding model performance are selected as all the sets of hyperparameter values ​​to be used when training the machine learning model using the sub-training sample set obtained from the next sampling. In response to the satisfaction of predetermined conditions, the training device 150 may stop sampling and use the highest-ranking hyperparameter values ​​among all the hyperparameter values ​​used when training the machine learning model with the sub-training sample set obtained from the last sampling as the hyperparameter values ​​that affect the training of the machine learning model.

[0103] As described above, after generating multiple sets of hyperparameter values ​​that affect the training of machine learning models, the training device 150 can train multiple machine learning models using the constructed machine learning samples based on each set of hyperparameter values. Then, it selects several machine learning models with the highest model performance from the trained machine learning models and fuses the selected machine learning models using a predetermined model fusion strategy to determine the final trained machine learning model.

[0104] Specifically, for model fusion, the training device 150 can use the plurality of machine learning models as an initial model candidate set, and perform the following model combination operation on the initial model candidate set: combine the machine learning model with the best performance among the plurality of machine learning models with the machine learning models in the initial model candidate set, and calculate the model performance of each combined machine learning model group; determine whether the model performance of each combined machine learning model group is improved compared with the best model performance before combination; if the model performance of the combined machine learning model group is not improved compared with the best model performance before combination, determine the machine learning model with the best performance before combination as the final machine learning model; if the model performance of the combined machine learning model group is improved compared with the best model performance before combination, add the combined machine learning model group with the best performance after combination to the initial model selection set to form the current model candidate set. Then, the training device 150 can repeat the same model combination operation as performed on the initial model candidate set a predetermined number of times on the current model candidate set, and select the machine learning model combination with the best performance from the finally generated current model candidate set. Finally, the training device 150 can determine the weights of the plurality of machine learning models during model fusion based on the number of each machine learning model included in the plurality of machine learning models in the machine learning model combination.

[0105] Optionally, if the number of training samples included in the training sample set is greater than the preset threshold, the training device 150 can obtain several sets of hyperparameter values ​​used when training the top-ranked machine learning models. Based on each set of hyperparameter values, the machine learning model is trained a second time using the training sample set. Then, the machine learning models that have undergone secondary training are fused using a predetermined model fusion strategy to determine the final trained machine learning model.

[0106] In addition, to further improve the performance of the final trained machine learning model, the training device 150 can obtain several sets of hyperparameter values ​​used when training the top-performing machine learning models, as well as the number of rounds of training the machine learning model using the training sample set based on each set of hyperparameter values. Based on each set of hyperparameter values, the machine learning model is trained for the corresponding number of rounds using all constructed machine learning samples to obtain several retrained machine learning models. The retrained machine learning models are then fused according to their respective weights during model fusion to obtain the final trained machine learning model.

[0107] In the above description, the machine learning model can be a neural network model or a logistic regression model, but is not limited to these.

[0108] Reference above Figure 1 The details mentioned when describing the various devices included in the automated model training system also apply here; therefore, for relevant details regarding the steps above, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions will not be repeated here.

[0109] The automatic model training method described above according to exemplary embodiments of this disclosure can automatically determine feature engineering strategies through meta-learning, generate sample features using the determined feature engineering strategies to construct machine learning samples, and train machine learning models using automatically obtained hyperparameters and the constructed machine learning samples. This greatly reduces the dependence of model training on the personal experience of machine learning experts and eliminates the need for relevant business personnel to have in-depth understanding of the business. It not only lowers the threshold for machine learning applications but also improves model training efficiency while ensuring training effectiveness.

[0110] Figure 4 A flowchart illustrating an example of a method for automatically training a machine learning model according to an exemplary embodiment.

[0111] To more intuitively understand the automatic model training method of this application, please refer to the following. Figure 4 This paper briefly describes an example of an automatic model training method, using a content recommendation scenario as an example.

[0112] Suppose a content provider wants to predict whether the content it recommends to users (e.g., news, ads, music, videos, etc.) will be accepted by users, then it can utilize... Figure 4 The automatic training method of the model is used to obtain the corresponding model to perform predictions.

[0113] Reference Figure 4First, in step S401, the computing device can obtain a dataset including various form data records from the content provider and can concatenate the form data to form a complete form. For example, each data record in the concatenated form may include multiple attribute fields (e.g., user ID, age, occupation, content ID, content topic name, keywords, login time, user logout time, user's historical favorites, etc.). Then, in step S402, the computing device can analyze the dataset to obtain meta-features reflecting the overall characteristics of the dataset and determine feature engineering strategies for the dataset based on these meta-features. For example, in step S403, the computing device can determine whether discrete features outnumber continuous features or whether the average fill factor of discrete features is higher than the average fill factor of continuous features in the data records. If so, in step S404, combined features are generated based on a logistic regression model. In step S405, the computing device can determine whether continuous features outnumber discrete features or whether the average fill factor of continuous features is higher than the average fill factor of discrete features in the data records. If so, features are generated based on an improved decision tree model in step S406. In step S407, the computing device determines whether there are time features in the data record. If so, in step S408, the data record is sorted according to the time features to generate a time sequence feature. In step S409, the computing device determines whether there are multiple time features in the data record. If so, in step S410, a feature is generated by subtracting the time features in pairs. (The above has already been referred to...) Figure 1 Various methods for generating features have been described and will not be repeated here. In step S411, the computing device may, for example, combine the generated features with the original features of the data record to construct a machine learning sample.

[0114] In step S412, the computing device determines whether the number of samples used to train the machine learning model (i.e., training samples in the training sample set, which may be a portion of all machine learning samples constructed in step S411) is greater than a threshold. If so, in step S413, the computing device samples the samples (e.g., density-based subsampling) to obtain a sub-training sample set; otherwise, proceed to step S414. Next, in step S414, the computing device generates multiple sets of hyperparameter values. Specifically, if the number of samples used to train the machine learning model is greater than the threshold, in step S414, using the sub-training sample set obtained in step S413, multiple sets of hyperparameter values ​​are generated using a combination of the random generation method and the Bayesian optimization model generation method mentioned above. If the number of samples used to train the machine learning model is less than the threshold, multiple sets of hyperparameter values ​​are randomly generated within the range of hyperparameter values ​​corresponding to that number. Subsequently, in step S415, the computing device trains the machine learning model based on each set of hyperparameter values ​​and obtains the model performance corresponding to each set of hyperparameter values. In step S416, the computing device determines whether the number of hyperparameter values ​​for which the corresponding model performance has been obtained has reached a threshold. If not, in step S417, new hyperparameters are generated. If so, in step S418, the top-ranked sets of hyperparameter values ​​with the corresponding model performance are selected from all hyperparameter values ​​for which the model performance has been obtained.

[0115] Next, in step S419, the computing device determines whether the number of samples used to train the machine learning model is greater than a threshold. If so, since the machine learning model was previously trained using a sub-training sample set, in step S420, the computing device trains the machine learning model using the entire training sample set based on each of the hyperparameter values ​​selected in step S418, thereby obtaining several machine learning models and the number of epochs for training each of the several machine learning models under each set of hyperparameter values. In step S421, the computing device determines the weights of the several machine learning models during model fusion. In step S422, the computing device trains the machine learning model for the corresponding number of epochs (i.e., the number of epochs obtained in step S420) using all the previously constructed machine learning samples, based on each of the selected hyperparameter values, and fuses the several machine learning models according to the weights of each machine learning model determined in step S421. Finally, in step S423, the computing device outputs the finally trained machine learning model. For example, the computing device can output the model to a content operator's local system or deploy it in the cloud. The output model can be used to provide corresponding prediction results for prediction samples. For instance, whenever a content operator wants to predict whether content it recommends to users will be accepted, it can construct corresponding prediction samples (constructing prediction samples in the same way as constructing machine learning samples described above) and input them into a trained machine learning model to obtain the corresponding prediction result (e.g., the probability that the content will be accepted by the user). The content operator can then make corresponding business decisions based on the prediction results, such as whether to recommend the content to users.

[0116] Here, the computing devices can be built by the content operator itself or provided by a dedicated entity that offers machine learning services. Accordingly, the computing devices can be located in the cloud (such as a public cloud, private cloud, or hybrid cloud) or on the content operator's on-premises system.

[0117] It should be noted that the above examples are merely examples of methods according to exemplary embodiments of this application, using content recommendation scenarios as an example. However, those skilled in the art will understand that the automatic model training method of this application is not limited to the above examples, and the exemplary embodiments of this application are not limited to content recommendation scenarios, but can be applied to any business scenario to help users make business decisions, such as credit card fraud detection, abnormal behavior detection, user interest extraction, and precision marketing.

[0118] The above has been referred to Figures 1 to 4An automatic model training system and method according to exemplary embodiments of the present disclosure are described. However, it should be understood that the apparatus and systems shown in the figures can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these systems and apparatuses may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these systems or apparatuses may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0119] Furthermore, the above method can be implemented by instructions recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of this application, a computer-readable storage medium storing instructions can be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device performs the following steps: acquiring a dataset comprising multiple data records; determining a feature engineering strategy for the dataset through meta-learning; generating features related to each data record in the dataset using the determined feature engineering strategy; constructing machine learning samples based at least on the generated features; and training a machine learning model using the constructed machine learning samples based on automatically obtained hyperparameter values.

[0120] The instructions stored in the aforementioned computer-readable storage medium can be executed in environments deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the instructions can also be used to perform additional steps beyond those described above, or to perform more specific processing while executing the above steps. The details of these additional steps and further processing are already provided in the reference... Figure 3 As mentioned in the description of machine learning methods, it will not be repeated here to avoid repetition.

[0121] It should be noted that the machine learning system according to the exemplary embodiments of this disclosure may rely entirely on the operation of computer programs or instructions to achieve the corresponding functions. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to achieve the corresponding functions.

[0122] On the other hand, when Figure 1 When the systems and apparatus shown are implemented as software, firmware, middleware, or microcode, the program code or code segment used to perform the corresponding operation can be stored in a computer-readable medium such as a storage medium, such that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0123] For example, according to an exemplary embodiment of this application, a system comprising at least one computing device and at least one storage instruction can be provided, wherein the instruction, when executed by the at least one computing device, causes the at least one computing device to perform the following steps: acquiring a dataset comprising multiple data records; determining a feature engineering strategy for the dataset through meta-learning; generating features related to each data record in the dataset using the determined feature engineering strategy; constructing machine learning samples based at least on the generated features; and training a machine learning model using the constructed machine learning samples based on automatically obtained hyperparameter values.

[0124] Specifically, the system described above can be deployed on a server or client, or on nodes in a distributed network environment. Furthermore, the system can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions. Additionally, the system may include a video display (such as a liquid crystal display) and a user interface (such as a keyboard, mouse, touch input device, etc.). Moreover, all components of the system can be interconnected via a bus and / or network.

[0125] Here, the system is not necessarily a single system, but can be a collection of any devices or circuits capable of executing the above instructions (or instruction sets) individually or in combination. The system can also be part of an integrated control system or system manager, or can be configured to interface with a portable electronic device locally or remotely (e.g., via wireless transmission).

[0126] In the system, the at least one computing device may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the at least one computing device may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc. The computing device may execute instructions or code stored in one of a storage devices, wherein the storage device may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transport protocol.

[0127] Storage devices can be integrated with computing devices, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, storage devices can include separate devices, such as external disk drives, storage arrays, or other storage devices usable by any database system. Storage devices and computing devices can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the computing device to read instructions stored in the storage device.

[0128] The foregoing has described various exemplary embodiments of this application. It should be understood that the above description is merely exemplary and not exhaustive, and this application is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for automatically training a machine learning model, performed by a computing device, comprising: Retrieve a dataset containing multiple data records; Meta-learning is used to determine feature engineering strategies for the dataset; Generate features relevant to each data record in the dataset using a defined feature engineering strategy; At least based on the generated features, machine learning samples are constructed; Based on automatically obtained hyperparameter values, a machine learning model is trained using constructed machine learning samples. The steps for determining feature engineering strategies for the dataset through meta-learning include: Analyze the dataset to obtain meta-features that reflect the overall characteristics of the dataset; Based on the obtained meta-features, a feature engineering strategy is determined for the dataset. The dataset involves image data, and the machine learning model is used to predict image categories.

2. The method as described in claim 1, wherein, The steps for determining a feature engineering strategy for the dataset based on the obtained meta-features include: Based on the correspondence between meta-features and feature engineering strategies, the feature engineering strategy for the dataset is automatically determined based on the obtained meta-features; or Based on the obtained meta-features, another machine learning model is used to determine the feature engineering strategy for the dataset.

3. The method as described in claim 2, wherein, The steps of determining a feature engineering strategy for the dataset using another machine learning model include: Based on the obtained meta-features, a clustering model, which serves as another machine learning model, is used to determine historical datasets similar to the dataset that already have corresponding model training results. The feature engineering strategy corresponding to the historical dataset is then determined as the feature engineering strategy for the dataset. Alternatively... The predicted sample based on the obtained meta-features is input as the prediction model of the other machine learning model, and the prediction result of the prediction model for the predicted sample is used as the feature engineering strategy for the dataset, wherein the prediction model is pre-trained to predict the feature engineering strategy corresponding to the meta-features based on the meta-features.

4. The method of claim 1, wherein, The steps involved in analyzing a dataset to obtain meta-features that reflect the overall characteristics of the dataset include: Analyze the dataset to obtain statistical information related to the number of data records included in the dataset, the types of features included in the data records, the number of features, and / or the values ​​of the features, as the meta-features.

5. The method of claim 4, wherein, Meta-features include at least one of the following: the number of data records in the dataset, the number of discrete features in the data records, the number of continuous features in the data records, whether there are time features in the data records, the number of time features in the data records, the fill factor of each discrete feature, the fill factor of each continuous feature, the average fill factor of all discrete features, the average fill factor of all continuous features, the mean, variance, median, maximum or minimum value of each continuous feature, and the number of distinct values ​​of each discrete feature.

6. The method of claim 5, wherein, The steps of determining a feature engineering strategy for the dataset based on the obtained meta-features include at least one of the following operations: In response to the fact that discrete features are stronger than continuous features in terms of distribution, the feature engineering strategy for the dataset is determined as follows: the continuous features are discretized to obtain discretized features, and the combination of discrete features that are beneficial to the model training effect is searched in the Cartesian product combination of the original discrete features and the discretized features as the generated features related to each data record. In response to the fact that continuous features are stronger than discrete features in distribution, the feature engineering strategy for the dataset is determined as follows: use the dataset to train a gradient boosting decision tree model, and use the leaf node number of each decision tree in which the sample formed by each data record falls as the generated feature related to each data record. In response to the presence of time features in the dataset, the feature engineering strategy for the dataset is determined as follows: sort the multiple data records according to time features, and in the process of training the logistic regression model sequentially using the sorted data records, obtain the time-series features corresponding to the current data record based on the parameter terms related to the model parameters of the logistic regression model, and use them as the generated features related to each data record. In response to the inclusion of multiple temporal features in the dataset, the feature engineering strategy for the dataset is determined as follows: for each data record, features related to each data record are generated by subtracting the sum of the multiple temporal features in pairs.

7. The method of claim 1, wherein, The steps for training a machine learning model using automatically obtained hyperparameter values ​​and constructed machine learning samples include: Use a portion of all the constructed machine learning samples as the training sample set; Based on the number of training samples included in the training sample set, different hyperparameter generation strategies are automatically adopted to generate multiple sets of hyperparameter values ​​that affect the training of the machine learning model. Based on each set of hyperparameter values, multiple machine learning models are trained using the constructed machine learning samples. The top-performing machine learning models are selected from among the trained models, and then fused together using a predetermined model fusion strategy to determine the final trained machine learning model.

8. The method of claim 7, wherein, The steps involved in automatically generating multiple sets of hyperparameter values ​​that influence the training of a machine learning model, based on the number of training samples included in the training sample set, and employing different hyperparameter generation strategies: In response to the fact that the number of training samples included in the training sample set is less than or equal to a preset threshold, multiple sets of hyperparameter values ​​are generated in the following ways: multiple sets of hyperparameter values ​​are randomly generated within the range of hyperparameter values ​​corresponding to the number of training samples; or, hyperparameter values ​​of some hyperparameters are predefined, and for the remaining hyperparameters, multiple sets of hyperparameter values ​​are randomly generated within the range of hyperparameter values ​​corresponding to the number of training samples. In response to the number of training samples in the training sample set being greater than the preset threshold, multiple sets of hyperparameter values ​​are generated in the following manner: Hyperparameter values ​​for some hyperparameters are predefined, and for the remaining hyperparameters, multiple sets of hyperparameter values ​​are generated using a combination of two methods: randomly generating hyperparameter values ​​within the range corresponding to the number of training samples and generating hyperparameter values ​​using a Bayesian optimization model. The Bayesian optimization model is constructed to reflect the relationship between hyperparameter values ​​and their corresponding model performance.

9. The method of claim 8, wherein, For the remaining hyperparameters, the steps for generating multiple sets of hyperparameter values ​​according to the described combination method include: A sub-training sample set is constructed by sampling training samples from the training sample set; For the remaining hyperparameters, a predetermined set of initial hyperparameter values ​​is randomly generated within the range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set. Based on each set of hyperparameter values, which consists of the predefined values ​​of some hyperparameters and each set of initial hyperparameter values ​​of the remaining hyperparameters, the machine learning model is trained using the sub-training sample set, and the model performance of the trained machine learning model is verified to obtain the model performance corresponding to each set of hyperparameter values. Based on the hyperparameter values ​​and their corresponding model effects, a Bayesian optimization model is established. The number of hyperparameter values ​​that still need to be generated for the remaining hyperparameters is continuously determined. A portion of this number is generated using a Bayesian optimization model, and another portion of this number is randomly generated within the range of hyperparameter values ​​corresponding to the number of training samples included in the sub-training sample set.

10. The method of claim 9, wherein, The steps of sampling training samples in the training sample set to construct a sub-training sample set include: sampling training samples in the training sample set multiple times to construct multiple sub-training sample sets with different numbers of training samples, wherein in the multiple sampling, the sub-training sample set obtained by the later sampling includes more training samples than the previous sampling. The step of generating multiple sets of hyperparameter values ​​according to the aforementioned combination method for the remaining hyperparameters further includes: Whenever new hyperparameter values ​​are generated for the remaining hyperparameters according to the combination method, the machine learning model is trained using the sub-training sample set obtained from the first sampling, based on each set of hyperparameter values ​​consisting of the predefined hyperparameter values ​​of the partial hyperparameters and the generated new hyperparameter values. When training a machine learning model using the sub-training sample set obtained from the previous sampling, select the group of hyperparameter values ​​with the highest model performance among all the hyperparameter values ​​used, and use them as the group of hyperparameter values ​​to be used when training a machine learning model using the sub-training sample set obtained from the next sampling. In response to the satisfaction of predetermined conditions, sampling is stopped, and the set of hyperparameter values ​​with the highest corresponding model performance among all the sets of hyperparameter values ​​used when training the machine learning model using the sub-training sample set obtained from the last sampling is used as the set of hyperparameter values ​​that affect the training of the machine learning model.

11. The method of claim 9, wherein, The steps for constructing a sub-training sample set include: constructing a sub-training sample set by performing density-based subsampling on the training samples in the training sample set.

12. The method of claim 11, wherein, The steps to construct a sub-training sample set by performing density-based subsampling on the training samples in the training sample set include: Reduce the dimensionality of the training samples in the training sample set to another dimensional space; A sub-training sample set is constructed by sampling the training samples based on the labels of the training samples and the distribution density of the training samples in the other dimension space.

13. The method of claim 12, wherein, The step of reducing the training sample set to the other dimensional space includes: mapping each training sample in the training sample set to a sample point on a two-dimensional plane; The step of constructing a sub-training sample set by sampling training samples based on the labels of the training samples and the distribution density of training samples in the other dimension space includes: Based on whether the training samples are positive or negative, classify all sample points on the two-dimensional plane into a set of positive sample points and a set of negative sample points. For each point set in the positive and negative sample point sets, sample points are sampled according to the set sampling ratio based on the distribution density of sample points in that point set, and the set consisting of training samples corresponding to the sampled sample points is used as a sub-training sample set.

14. The method of claim 13, wherein, The steps for sampling points according to the distribution density of the sample points in the point set and according to the set sampling ratio include: Based on the maximum and minimum values ​​of the x and y coordinates of all sample points in the point set on the two-dimensional plane, determine the planar region covering all sample points in the point set. The defined planar region is divided into multiple sub-regions on an equal basis; Count the number of sub-regions into which the sample points fall; If the number of sub-regions in which sample points fall exceeds the preset number, random sampling is performed in each sub-region according to the set sampling ratio. In response to the number of sub-regions in which sample points fall is less than or equal to the preset number, the number of sub-regions is increased until the number of sub-regions in which sample points fall is greater than the preset number. Then, random sampling is performed in each sub-region according to the set sampling ratio.

15. The method of claim 9, wherein, For the remaining hyperparameters, the step of generating multiple sets of hyperparameter values ​​according to the described combination method further includes: After the Bayesian optimization model is established, it is updated whenever new hyperparameter values ​​and their corresponding model effects are obtained.

16. The method of claim 9, wherein, When the number of training samples in the training sample set is greater than the preset threshold, the steps for determining the finally trained machine learning model include: Obtain several sets of hyperparameter values ​​used when training the top-performing machine learning models; Based on each set of hyperparameter values ​​from the aforementioned sets, the machine learning model is trained a second time using the training sample set. The final trained machine learning model is determined by fusing several machine learning models that have undergone secondary training using a predetermined model fusion strategy.

17. The method of claim 7, wherein, The steps to determine the final trained machine learning model include: The aforementioned machine learning models are used as an initial model candidate set; For the initial model candidate set, the following model combination operation is performed: the machine learning model with the best performance among the plurality of machine learning models is combined with the machine learning models in the initial model candidate set, and the model performance of each combined machine learning model group is calculated; it is determined whether the model performance of each combined machine learning model group is improved compared with the best model performance before combination; if the model performance of the combined machine learning model group is not improved compared with the best model performance before combination, the machine learning model with the best performance before combination is determined as the final machine learning model; if the model performance of the combined machine learning model group is improved compared with the best model performance before combination, the combined machine learning model group with the best performance after combination is added to the initial model selection set to form the current model candidate set. The same model combination operation as the one performed on the initial model candidate set is repeated a predetermined number of times for the current model candidate set, and the machine learning model combination with the best model performance is selected from the final generated current model candidate set. The weights of each of the several machine learning models included in the machine learning model combination are determined during model fusion.

18. The method of claim 17, wherein the step of determining the finally trained machine learning model further comprises: The set of hyperparameter values ​​used when obtaining the top-ranked machine learning models, and the number of rounds of training the machine learning model using the training sample set based on each of the set of hyperparameter values; Based on the hyperparameter values ​​for each set, the machine learning model is trained for the corresponding number of rounds using all the constructed machine learning samples to obtain several retrained machine learning models. Several retrained machine learning models are fused according to their respective weights during model fusion to obtain the final trained machine learning model.

19. The method of claim 6, wherein, The machine learning model is either a neural network model or a logistic regression model.

20. The method of claim 9, wherein, The operation of generating a portion of the set of numbers using a Bayesian optimization model is performed in parallel with the operation of randomly generating another portion of the set of numbers.

21. A computer-readable storage medium for storing instructions, wherein, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 20.

22. A system comprising at least one computing device and at least one storage device for storing instructions, wherein, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 20.

23. A system for automatically training machine learning models, comprising: A dataset acquisition device is configured to acquire a dataset that includes multiple data records; A feature engineering strategy determination apparatus is configured to determine a feature engineering strategy for the dataset through meta-learning. The feature generation device is configured to generate features related to each data record in the dataset using a defined feature engineering strategy. The sample construction apparatus is configured to construct machine learning samples based at least on the generated features; The training device is configured to train a machine learning model using constructed machine learning samples based on automatically obtained hyperparameter values. The feature engineering strategy determination device is configured to: analyze a dataset to obtain meta-features reflecting the overall characteristics of the dataset; and determine a feature engineering strategy for the dataset based on the obtained meta-features. The dataset involves image data, and the machine learning model is used to predict image categories.

Images