A method, device and system for implementing automatic hyperparameter selection

Abstract

The application provides a method, device and system for realizing automatic hyperparameter selection, the method comprising: obtaining a sample set, wherein each sample in the sample set comprises corresponding data and a label; splitting the sample set into a training set and a validation set according to a preset splitting rule; performing hyperparameter search according to a preset parameter tuning algorithm, and training a machine learning model based on the training set to obtain multiple groups of candidate hyperparameter values; verifying the machine learning model corresponding to each group of candidate hyperparameter values based on the validation set to obtain a verification score of the machine learning model corresponding to each group of candidate hyperparameter values on the validation set; and selecting at least one group of preferred hyperparameter values from the multiple groups of candidate hyperparameter values according to the verification score.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically, to a method for implementing automatic hyperparameter selection, an apparatus for implementing automatic hyperparameter selection, a system including at least one computing device and at least one storage device, and a readable storage medium. Background Technology

[0002] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are found in all areas of artificial intelligence.

[0003] Before training a traditional machine learning model, hyperparameters are selected using hyperparameter tuning algorithms. Typically, hyperparameters need to be optimized to choose an optimal set for the model, thereby improving learning performance and effectiveness.

[0004] In real-world projects, the cost of deploying machine learning models is significant, making it crucial to select stable hyperparameters for these models. Summary of the Invention

[0005] One objective of this invention is to provide a new technical solution for automatically selecting hyperparameters of machine learning models.

[0006] According to a first aspect of the present invention, a method for implementing automatic hyperparameter selection is provided, comprising:

[0007] Obtain a sample set, wherein each sample in the sample set includes corresponding data and labels;

[0008] The sample set is split into a training set and a validation set according to a preset splitting rule;

[0009] The hyperparameter search is performed according to the preset hyperparameter tuning algorithm, and the machine learning model is trained based on the training set to obtain multiple sets of candidate hyperparameter values.

[0010] Based on the validation set, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set.

[0011] Based on the verification score, at least one set of preferred hyperparameter values ​​is selected from the plurality of candidate hyperparameter values.

[0012] Optionally, splitting the sample set into a training set and a validation set according to a preset splitting rule includes:

[0013] Determine whether the labels in the sample set have a temporal relationship;

[0014] Obtain the splitting rule corresponding to the judgment result, and split the sample set into a training set and a validation set according to the splitting rule.

[0015] Optionally, obtaining the splitting rule corresponding to the judgment result, and splitting the sample set into a training set and a validation set according to the splitting rule includes:

[0016] Given that the labels in the sample set have a temporal relationship, obtain the time split ratio of the training set and the validation set;

[0017] Select the target time window with a set duration;

[0018] The target time window is divided into a first part of the time window for selecting the training set and a second part of the time window for selecting the validation set according to the time split ratio.

[0019] The samples whose generation time corresponds to the first part of the time window are selected as the training set, and the samples whose generation time corresponds to the second part of the time window are selected as the validation set.

[0020] Optional, also includes:

[0021] Detect whether the event that initiates self-learning has occurred;

[0022] In the event that the event occurs, a new training set and a new validation set are obtained;

[0023] The machine learning model corresponding to each set of candidate hyperparameter values ​​is iteratively trained based on the new training set.

[0024] The iterative machine learning model corresponding to each set of candidate hyperparameter values ​​is validated based on the new validation set to obtain a new validation score for the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​on the new validation set.

[0025] Based on the new verification score, preferred hyperparameter values ​​are reselected from the multiple sets of candidate hyperparameter values.

[0026] Optionally, obtaining the new training set and the new validation set includes:

[0027] Get the preset sliding duration;

[0028] The target time window is slid according to the specified sliding duration;

[0029] The samples corresponding to the generation time and the first part of the sliding time window are selected as the new training set, and the samples corresponding to the generation time and the second part of the sliding time window are selected as the new validation set.

[0030] Optionally, the event includes at least one of the following:

[0031] The preset startup cycle has been reached;

[0032] The preset startup time has been reached;

[0033] The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set has been completed.

[0034] Received the specified operation executed by the user.

[0035] Optionally, obtaining the splitting rule corresponding to the judgment result, and splitting the sample set into a training set and a validation set according to the splitting rule includes:

[0036] When the labels in the sample set do not have a temporal relationship, a preset quantity splitting ratio is obtained;

[0037] Based on the quantity splitting ratio and the number of samples in the sample set, a first quantity for determining the training set and a second quantity for determining the validation set are obtained;

[0038] A first number of samples are selected from the sample set to obtain the training set; a second number of samples are selected to obtain the validation set.

[0039] Optionally, determining whether the labels of the sample set have a temporal relationship includes:

[0040] Obtain the application scenarios of the machine learning model;

[0041] When the application scenario is a specified application scenario, it is determined that the labels of the sample set have a temporal relationship;

[0042] or,

[0043] Obtain the type of data content in the sample set;

[0044] When the data content is of a specified type, it is determined that the labels of the sample set have a temporal relationship.

[0045] Optionally, when the labels of the samples in the sample set are continuous values, the method further includes:

[0046] The labels are discretized.

[0047] Optionally, the step of searching for hyperparameters according to a preset hyperparameter tuning algorithm and training a machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values ​​includes:

[0048] Multiple rounds of training are performed, wherein in each round of training, one or more sets of initial hyperparameter values ​​are obtained by hyperparameter search according to the hyperparameter tuning algorithm, one or more corresponding machine learning models are trained based on the training set and the one or more sets of initial hyperparameter values, and the corresponding model scores are obtained.

[0049] Based on the model scores obtained from the multiple rounds of training, multiple sets of candidate hyperparameter values ​​are selected.

[0050] Optionally, the step of selecting multiple sets of candidate hyperparameter values ​​based on the model scores obtained in the multiple rounds of training includes:

[0051] Select initial hyperparameter values ​​whose training scores exceed a preset first score threshold as candidate hyperparameter values; or,

[0052] The initial hyperparameter values ​​are sorted in descending order based on the training scores, and the first sorting value of each initial hyperparameter value is obtained.

[0053] The initial hyperparameter value within a first set range of the first sorting value is selected as the candidate hyperparameter value.

[0054] Optional, also includes:

[0055] In response to a request to interrupt the search for hyperparameter values, the search for multiple sets of initial hyperparameter values ​​for the machine learning model is interrupted.

[0056] In response to the request to continue searching for hyperparameter values, the search continues for multiple sets of initial hyperparameter values ​​for the machine learning model.

[0057] Optionally, the parameter tuning algorithm is set in a preset infinite loop program.

[0058] The method further includes:

[0059] The initial hyperparameter values ​​obtained each time according to the parameter tuning algorithm are saved to the history.

[0060] In response to a request to continue searching for hyperparameter values, continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model includes:

[0061] In response to a request to continue searching for hyperparameter values, the history is loaded, and the search for the next set of initial hyperparameter values ​​for the machine learning model is continued based on the history.

[0062] Optionally, the number of verification sets can be multiple.

[0063] The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set, and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set, includes:

[0064] For each of the aforementioned validation sets, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the corresponding validation set.

[0065] The step of selecting preferred hyperparameter values ​​based on the verification score includes:

[0066] For each set of candidate hyperparameter values, determine the comprehensive validation score of the corresponding machine learning model on all validation sets;

[0067] Select the optimal hyperparameter values ​​based on the comprehensive verification score;

[0068] The comprehensive verification score includes any one of the following: average, median, minimum, and maximum.

[0069] Optionally, selecting at least one preferred set of hyperparameter values ​​from the plurality of candidate hyperparameter values ​​based on the verification score includes:

[0070] Candidate hyperparameter values ​​whose verification scores exceed a preset second score threshold are selected as the preferred hyperparameter values;

[0071] or,

[0072] The candidate hyperparameter values ​​are sorted in descending order based on the verification scores, and a second sorting value is obtained for each group of candidate hyperparameter values.

[0073] The candidate hyperparameter values ​​within a second set range of the second sorting value are selected as the preferred hyperparameter values.

[0074] Optionally, the parameter tuning algorithm includes any one or more of the following: grid search algorithm, random search algorithm, and Bayesian search algorithm.

[0075] Optional, also includes:

[0076] Provides a settings entry point for configuring the application scenarios of the machine learning model;

[0077] Obtain the application scenario input through the settings entry point;

[0078] The sample set is obtained based on the application scenario input.

[0079] Optional, also includes:

[0080] The preferred hyperparameter values ​​are shown.

[0081] Optional, also includes:

[0082] Provide an entry point for inputting the splitting rules;

[0083] The splitting rules are obtained through the aforementioned entry point.

[0084] Optional, also includes:

[0085] Find the application scenarios to which the machine learning model is applicable;

[0086] Find application projects that match the described application scenario;

[0087] The machine learning model corresponding to the selected hyperparameter values ​​after training is input into the application project.

[0088] According to a second aspect of the present invention, an apparatus for implementing automatic hyperparameter selection is provided, comprising:

[0089] The sample acquisition module is used to acquire a sample set, wherein each sample in the sample set includes corresponding data and labels;

[0090] The sample splitting module is used to split the sample set into a training set and a validation set according to a preset splitting rule;

[0091] The hyperparameter tuning search module is used to search for hyperparameters according to a preset hyperparameter tuning algorithm and to train a machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values.

[0092] The scoring determination module is used to verify the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the verification set, and to obtain the verification score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the verification set.

[0093] The hyperparameter selection module is used to select at least one set of preferred hyperparameter values ​​from the plurality of candidate hyperparameter values ​​based on the verification score.

[0094] Optionally, the sample splitting module is used for:

[0095] Determine whether the labels in the sample set have a temporal relationship;

[0096] Obtain the splitting rule corresponding to the judgment result, and split the sample set into a training set and a validation set according to the splitting rule.

[0097] Optionally, obtaining the splitting rule corresponding to the judgment result, and splitting the sample set into a training set and a validation set according to the splitting rule includes:

[0098] Given that the labels in the sample set have a temporal relationship, obtain the time split ratio of the training set and the validation set;

[0099] Select the target time window with a set duration;

[0100] The target time window is divided into a first part of the time window for selecting the training set and a second part of the time window for selecting the validation set according to the time split ratio.

[0101] The samples whose generation time corresponds to the first part of the time window are selected as the training set, and the samples whose generation time corresponds to the second part of the time window are selected as the validation set.

[0102] Optional, also includes:

[0103] The self-learning detection module is used to detect whether the event that initiates self-learning has occurred;

[0104] A new sample acquisition module is used to acquire a new training set and a new validation set in the event said event occurs;

[0105] The iterative training module is used to iteratively train the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new training set.

[0106] The new score determination module is used to verify the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new validation set, and to obtain a new validation score of the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​on the new validation set.

[0107] A new hyperparameter selection module is used to reselect preferred hyperparameter values ​​from the multiple sets of candidate hyperparameter values ​​based on the new verification score.

[0108] Optionally, obtaining the new training set and the new validation set includes:

[0109] Get the preset sliding duration;

[0110] The target time window is slid according to the specified sliding duration;

[0111] The samples corresponding to the generation time and the first part of the sliding time window are selected as the new training set, and the samples corresponding to the generation time and the second part of the sliding time window are selected as the new validation set.

[0112] Optionally, the event includes at least one of the following:

[0113] The preset startup cycle has been reached;

[0114] The preset startup time has been reached;

[0115] The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set has been completed.

[0116] Received the specified operation executed by the user.

[0117] Optionally, obtaining the splitting rule corresponding to the judgment result, and splitting the sample set into a training set and a validation set according to the splitting rule includes:

[0118] When the labels in the sample set do not have a temporal relationship, a preset quantity splitting ratio is obtained;

[0119] Based on the quantity splitting ratio and the number of samples in the sample set, a first quantity for determining the training set and a second quantity for determining the validation set are obtained;

[0120] A first number of samples are selected from the sample set to obtain the training set; a second number of samples are selected to obtain the validation set.

[0121] Optionally, determining whether the labels of the sample set have a temporal relationship includes:

[0122] Obtain the application scenarios of the machine learning model;

[0123] When the application scenario is a specified application scenario, it is determined that the labels of the sample set have a temporal relationship;

[0124] or,

[0125] Obtain the type of data content in the sample set;

[0126] When the data content is of a specified type, it is determined that the labels of the sample set have a temporal relationship.

[0127] Optionally, when the labels of the samples in the sample set are continuous values, the apparatus further includes:

[0128] A module for discretizing the labels.

[0129] Optionally, the parameter tuning search module is used for:

[0130] Multiple rounds of training are performed, wherein in each round of training, one or more sets of initial hyperparameter values ​​are obtained by hyperparameter search according to the hyperparameter tuning algorithm, one or more corresponding machine learning models are trained based on the training set and the one or more sets of initial hyperparameter values, and the corresponding model scores are obtained.

[0131] Based on the model scores obtained from the multiple rounds of training, multiple sets of candidate hyperparameter values ​​are selected.

[0132] Optionally, the step of selecting multiple sets of candidate hyperparameter values ​​based on the model scores obtained in the multiple rounds of training includes:

[0133] Select initial hyperparameter values ​​whose training scores exceed a preset first score threshold as candidate hyperparameter values; or,

[0134] The initial hyperparameter values ​​are sorted in descending order based on the training scores, and the first sorting value of each initial hyperparameter value is obtained.

[0135] The initial hyperparameter value within a first set range of the first sorting value is selected as the candidate hyperparameter value.

[0136] Optional, also includes:

[0137] A module for interrupting the search for multiple sets of initial hyperparameter values ​​of the machine learning model in response to a request to interrupt the search for hyperparameter values;

[0138] A module for continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model in response to a request to continue searching for hyperparameter values.

[0139] Optionally, the parameter tuning algorithm is set in a preset infinite loop program.

[0140] The device further includes:

[0141] This module is used to save the initial hyperparameter values ​​obtained each time according to the parameter tuning algorithm to the history.

[0142] In response to a request to continue searching for hyperparameter values, continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model includes:

[0143] In response to a request to continue searching for hyperparameter values, the history is loaded, and the search for the next set of initial hyperparameter values ​​for the machine learning model is continued based on the history.

[0144] Optionally, the number of verification sets can be multiple.

[0145] The score determination module is used for:

[0146] For each of the aforementioned validation sets, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the corresponding validation set.

[0147] The step of selecting preferred hyperparameter values ​​based on the verification score includes:

[0148] For each set of candidate hyperparameter values, determine the comprehensive validation score of the corresponding machine learning model on all validation sets;

[0149] Select the optimal hyperparameter values ​​based on the comprehensive verification score;

[0150] The comprehensive verification score includes any one of the following: average, median, minimum, and maximum.

[0151] Optionally, the hyperparameter selection module is used for:

[0152] Candidate hyperparameter values ​​whose verification scores exceed a preset second score threshold are selected as the preferred hyperparameter values;

[0153] or,

[0154] The candidate hyperparameter values ​​are sorted in descending order based on the verification scores, and a second sorting value is obtained for each group of candidate hyperparameter values.

[0155] The candidate hyperparameter values ​​within a second set range of the second sorting value are selected as the preferred hyperparameter values.

[0156] Optionally, the parameter tuning algorithm includes any one or more of the following: grid search algorithm, random search algorithm, and Bayesian search algorithm.

[0157] Optional, also includes:

[0158] A module for providing a settings entry point for configuring the application scenarios of the machine learning model;

[0159] A module for obtaining the application scenario input through the setting entry;

[0160] A module for obtaining the sample set based on the application scenario input.

[0161] Optional, also includes:

[0162] Module used to display the preferred hyperparameter values.

[0163] Optional, also includes:

[0164] A module for providing an entry point for inputting the splitting rules;

[0165] A module used to obtain the splitting rules through the entry point.

[0166] Optional, also includes:

[0167] Use the module to obtain the application scenarios to which the machine learning model is applicable;

[0168] A module for finding application projects that match the application scenario;

[0169] This module is used to input the machine learning model corresponding to the trained preferred hyperparameter values ​​into the application project.

[0170] According to a third aspect of the present invention, a system is provided comprising at least one computing device and at least one storage device, wherein the at least one storage device is configured to store instructions for controlling the at least one computing device to perform the method according to the first aspect of the present invention.

[0171] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention.

[0172] One beneficial effect of this invention is that by selecting candidate hyperparameter values ​​from hyperparameter values ​​obtained through hyperparameter search according to a preset hyperparameter tuning algorithm using a training set, and then selecting preferred hyperparameter values ​​from the candidate hyperparameter values ​​using a validation set, the preferred hyperparameter values ​​obtained through the second selection can be made more stable. In turn, the machine learning model corresponding to the preferred hyperparameter values ​​can also be made more stable, thereby improving the performance and effect of the machine learning model corresponding to the preferred hyperparameter values.

[0173] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0174] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0175] Figure 1 This is a block diagram illustrating an example of a hardware configuration for an electronic device that can be used to implement embodiments of the present invention.

[0176] Figure 2 This is a flowchart illustrating a method for implementing automatic hyperparameter selection according to an embodiment of the present invention;

[0177] Figure 3 This is a flowchart illustrating the self-learning steps according to an embodiment of the present invention;

[0178] Figure 4 This is a schematic diagram of a segmented sample set according to an embodiment of the present invention;

[0179] Figure 5 This is a block diagram of a device for implementing automatic hyperparameter selection according to an embodiment of the present invention;

[0180] Figure 6 This is a block diagram of a system according to an embodiment of the present invention. Detailed Implementation

[0181] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0182] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0183] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0184] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0185] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0186] Hereinafter, various embodiments and examples according to the present invention will be described with reference to the accompanying drawings.

[0187] <Hardware Configuration>

[0188] Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device 1000 that can implement embodiments of the present invention.

[0189] Electronic device 1000 can be a portable computer, desktop computer, mobile phone, tablet computer, etc. For example... Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication, specifically including Wi-Fi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, etc. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, motion input, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800.

[0190] Figure 1 The electronic devices shown are merely illustrative and in no way intended to limit the invention, its application, or use. In embodiments applied to the invention, the memory 1200 of the electronic device 1000 is used to store instructions for controlling the processor 1100 to operate to execute any of the methods for implementing automatic hyperparameter selection provided in embodiments of the invention. Those skilled in the art will understand that, although... Figure 1 The electronic device 1000 is shown with multiple devices shown; however, the present invention may refer only to some of these devices. For example, the electronic device 1000 may refer only to the processor 1100 and the storage device 1200. Those skilled in the art can design instructions according to the disclosed scheme of the present invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0191] <Method Example 1>

[0192] In this embodiment, a method for implementing automatic hyperparameter selection is provided. This method for implementing automatic hyperparameter selection can be implemented by an electronic device. The electronic device can be, for example,... Figure 1 The electronic device shown is 1000.

[0193] according to Figure 2 As shown, the method for implementing automatic hyperparameter selection in this embodiment may include the following steps S2100 to S2500:

[0194] Step S2100: Obtain the sample set.

[0195] Each sample in the sample set includes corresponding data and labels.

[0196] Step S2200: The sample set is split into a training set and a validation set according to the preset splitting rules.

[0197] Step S2300: Perform hyperparameter search according to the preset hyperparameter tuning algorithm, and train the machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values.

[0198] In one embodiment of the present invention, the parameter tuning algorithm may be pre-set according to the application scenario or specific requirements. The parameter tuning algorithm includes any one or more of the following: grid search algorithm, random search algorithm, and Bayesian search algorithm.

[0199] Step S2400: Validate the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set, and obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set.

[0200] Step S2500: Select at least one set of preferred hyperparameter values ​​from multiple sets of candidate hyperparameter values ​​based on the verification score.

[0201] In embodiments of the present invention, candidate hyperparameter values ​​are selected from hyperparameter values ​​obtained by hyperparameter search according to a preset hyperparameter tuning algorithm through a training set, and preferred hyperparameter values ​​are selected from the candidate hyperparameter values ​​through a validation set. This makes the preferred hyperparameter values ​​obtained through the second selection more stable, thereby making the machine learning model corresponding to the preferred hyperparameter values ​​more stable and improving the performance and effect of the machine learning model corresponding to the preferred hyperparameter values.

[0202] Typically, labels in classification scenarios are discrete values, while labels in regression scenarios are continuous values. For example, when scoring an image, the score is a continuous value between 0 and 5, meaning scores like 4.351 and 0.24 might appear.

[0203] In one embodiment of the present invention, when the labels of the samples in the sample set are continuous values, the method may further include:

[0204] Discretize the labels.

[0205] In one embodiment of the present invention, the labels may be discretized by using equidistant bins.

[0206] For example, based on the existing interval span, it can be divided into 5 buckets: bucket 1 represents 0 to 1, bucket 2 represents 1 to 2, bucket 3 represents 2 to 3, bucket 4 represents 3 to 4, and bucket 5 represents 4 to 5. The bucket number corresponding to the score falling into the interval is marked, and the bucket number is used as the new label. The new label is the discrete value.

[0207] In one embodiment of the present invention, the splitting rules may be pre-defined according to the application scenario or specific requirements.

[0208] In one embodiment of the present invention, the method may further include:

[0209] Provides an entry point for inputting splitting rules; the splitting rules are retrieved through this entry point.

[0210] In this embodiment, users can input splitting rules through this entry point.

[0211] In one embodiment of the present invention, the method may further include:

[0212] Provides a settings entry point for configuring application scenarios for machine learning models;

[0213] Identify the application scenarios that can be accessed through the entry point settings;

[0214] Obtain a sample set based on the input application scenario.

[0215] Specifically, the electronic device executing the embodiments of the present invention may have a sample set corresponding to multiple application scenarios pre-stored in it. Based on the application scenario input through the provided application scenario setting entry, the sample set matching the application scenario is obtained for machine learning training. In this way, the machine learning model corresponding to the preferred hyperparameter values ​​can be applied to the input application scenario.

[0216] In one embodiment of the present invention, the method may further include: displaying preferred hyperparameter values. Specifically, the preferred hyperparameter values ​​may be displayed in a display interface for user reference.

[0217] In one embodiment of the present invention, the method may further include:

[0218] Identify the application scenarios for which machine learning models are applicable;

[0219] Find application projects that match this application scenario;

[0220] The machine learning model corresponding to the optimized hyperparameter values ​​after training is input into the application project.

[0221] In this embodiment, the machine learning model corresponding to the optimized hyperparameter values ​​after training is input into an application project that matches the application scenario to which it is applicable, so that the machine learning model can be used to process the sample information in the corresponding application project.

[0222] <Method Example 2>

[0223] Based on the aforementioned method embodiment one, performing hyperparameter search according to a preset hyperparameter tuning algorithm and training a machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values ​​may include the following steps S2310 to S2320:

[0224] Step S2310: Perform multiple rounds of training. In each round of training, a set of one or more initial hyperparameter values ​​are obtained by hyperparameter search according to the hyperparameter tuning algorithm. One or more corresponding machine learning models are trained based on the training set and the set of one or more initial hyperparameter values, and the corresponding model score is obtained.

[0225] In one embodiment of the present invention, the method may further include:

[0226] In response to a request to interrupt the search for hyperparameter values, the search for multiple sets of initial hyperparameter values ​​for the machine learning model is interrupted.

[0227] In response to the request to continue searching for hyperparameter values, continue searching for multiple sets of initial hyperparameter values ​​for the machine learning model.

[0228] In this embodiment, the requests to interrupt the search for hyperparameter values ​​and to continue the search for hyperparameter values ​​can be triggered by the user based on the application scenario or specific needs. If a request to interrupt the search for hyperparameter values ​​is received after obtaining one or more initial hyperparameter values ​​through hyperparameter search using a hyperparameter tuning algorithm, and no corresponding model score has been obtained, then the steps of training one or more corresponding machine learning models based on the training set and one or more initial hyperparameter values, and obtaining the corresponding model score will continue. Once a model score is obtained, the search for initial hyperparameter values ​​will no longer continue.

[0229] In one embodiment of the present invention, the hyperparameter tuning framework may support resuming interrupted runs. Therefore, the user can either directly send a request to interrupt the search for hyperparameter values, thus interrupting the search for multiple sets of initial hyperparameter values ​​for the machine learning model; or, the user can directly send a request to continue the search for hyperparameter values, thus continuing the search for multiple sets of initial hyperparameter values ​​for the machine learning model.

[0230] In one embodiment of the present invention, if the hyperparameter tuning framework does not support breakpoint resumption, such as the hyperopt library that supports Bayesian optimization, the hyperparameter tuning algorithm can be set in a preset infinite loop program. In this case, the method may further include: saving the initial hyperparameter values ​​searched by the hyperparameter tuning algorithm each time to the history record.

[0231] Based on this, in response to a request to continue searching for hyperparameter values, continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model may include: in response to a request to continue searching for hyperparameter values, loading the history and continuing to search for the next set of initial hyperparameter values ​​for the machine learning model based on the history.

[0232] In this embodiment, the hyperparameter tuning algorithm is set within a preset infinite loop, searching for only one set of hyperparameters at a time and saving the initial hyperparameter values ​​found each time to the history. If a user requests to interrupt the search for hyperparameter values, the search is terminated; otherwise, the search continues. If the user requests to continue searching for hyperparameter values ​​after the search has stopped, the history can be loaded, and the search can continue for the next set of initial hyperparameter values. In this way, the hyperparameter tuning algorithm can also resume from where it left off.

[0233] Step S2320: Select multiple sets of candidate hyperparameter values ​​based on the model scores obtained from multiple rounds of training.

[0234] In one embodiment of the present invention, selecting multiple sets of candidate hyperparameter values ​​based on model scores obtained from multiple rounds of training may include:

[0235] Initial hyperparameter values ​​that exceed a preset first score threshold are selected as candidate hyperparameter values.

[0236] The first score threshold can be set in advance according to the application scenario or specific needs. For example, the first score threshold can be 60.

[0237] In another embodiment of the present invention, selecting multiple sets of candidate hyperparameter values ​​based on model scores obtained from multiple rounds of training may further include:

[0238] The initial hyperparameter values ​​are sorted in descending order based on the training scores, and the first sorted value of each initial hyperparameter value is obtained. The initial hyperparameter values ​​within a first set range of the first sorted value are selected as candidate hyperparameter values.

[0239] The first set range can be pre-defined according to the application scenario or specific requirements. For example, the first set range can be [1, 3]. Then, the initial hyperparameter values ​​with the first sorting value of 1, 2, and 3 can be selected as candidate hyperparameter values.

[0240] In one embodiment of the present invention, selecting at least one set of preferred hyperparameter values ​​from multiple sets of candidate hyperparameter values ​​based on the verification score may include:

[0241] Candidate hyperparameter values ​​whose verification scores exceed a preset second score threshold are selected as preferred hyperparameter values.

[0242] The second score threshold can be set in advance according to the application scenario or specific needs. For example, the second score threshold can be 90. Then, candidate hyperparameter values ​​with a verification score of more than 90 can be selected as preferred hyperparameter values.

[0243] In another embodiment of the present invention, selecting at least one set of preferred hyperparameter values ​​from a plurality of candidate hyperparameter values ​​based on the verification score may include:

[0244] The candidate hyperparameter values ​​are sorted in descending order based on the verification scores, and a second sorting value is obtained for each group of candidate hyperparameter values. The candidate hyperparameter values ​​with the second sorting value within a second set range are selected as the preferred hyperparameter values.

[0245] The second setting range can be pre-defined according to the application scenario or specific requirements. For example, the second setting range can be [1, 1], then the candidate hyperparameter value with the first sorting value of 1 can be selected as the preferred hyperparameter value.

[0246] <Method Example 3>

[0247] Based on the aforementioned method embodiment one, splitting the sample set into a training set and a validation set according to a preset splitting rule may include the following steps S2210 to S2220:

[0248] Step S2210: Determine whether the labels in the sample set have a temporal relationship.

[0249] If the labels of a sample depend on the time when the data was acquired, then the labels of the sample set can be considered to have a temporal relationship. If the labels of a sample depend on the structure of the data itself, then the labels of the sample set can be considered not to have a temporal relationship.

[0250] For example, in image data, whether a picture is classified as a cat or a dog does not depend on the order in which it was taken, but rather on the structure of the data itself. Therefore, image classification labels do not have a temporal relationship. However, in anti-fraud scenarios, labels indicating whether a user is fraudulent are temporal because these labels are based on a user's transaction behavior over a period of time. A user may be a legitimate user for a long period, but may engage in fraudulent behavior at a certain point in time.

[0251] In one embodiment of the present invention, determining whether the labels of a sample set have a temporal relationship may include:

[0252] Obtain the application scenario of the machine learning model; if the application scenario is specified, determine that the labels of the sample set have a temporal relationship.

[0253] The specified application scenario can be a pre-defined application scenario based on specific needs. For example, the specified application scenario could be an anti-fraud scenario.

[0254] In anti-fraud scenarios, as time goes on, fraudulent users may discover that a certain behavior pattern will be triggered by the anti-fraud system, and may then adopt new criminal patterns to evade detection. Therefore, the labels of the sample set in anti-fraud scenarios have a temporal relationship.

[0255] In another embodiment of the present invention, determining whether the labels of the sample set have a temporal relationship may include:

[0256] Obtain the type of data content in the sample set; if the type of data content is a specified type, determine that the labels in the sample set have a temporal relationship.

[0257] The specified type can be pre-defined based on the application scenario or specific requirements. For example, the specified type could be a transaction type. Therefore, if the data content in the sample set is of the transaction type, the fraud tags in the sample set will have a temporal relationship.

[0258] Step S2220: Obtain the splitting rule corresponding to the judgment result, and split the sample set into a training set and a validation set according to the splitting rule.

[0259] In one embodiment of the present invention, obtaining the splitting rule corresponding to the judgment result and splitting the sample set into a training set and a validation set according to the splitting rule may include the following steps S2221a to S2224a:

[0260] Step S2221a: Given that the labels in the sample set have a temporal relationship, obtain the time split ratio of the training set and the validation set.

[0261] Specifically, the time split ratio can be obtained from the splitting rules. If the splitting rule is 8:1:1, the time split ratio can be 8:1. If the splitting rule is 6:1:1:1:1, the time split ratio can be 6:1.

[0262] Step S2222a: Select a target time window with a set time length.

[0263] In one embodiment of the present invention, the target time window can be preset according to the application scenario or specific needs. For example, the preset time window can be 9 months.

[0264] Step S2223a: The target time window is divided into a first part of the time window for selecting the training set and a second part of the time window for selecting the validation set according to the time split ratio.

[0265] The first part of the time window is located before the second part of the time window.

[0266] In one embodiment of the present invention, the lengths of the first and second time windows can be determined based on a time-splitting ratio and a set time length. The ratio of the length of the first to the length of the second time window conforms to the time-splitting ratio.

[0267] For example, if the time split ratio is 6:1 and the target time window is 7 months long, the first 6 months of the target time window can be used as the first part of the time window, and the last part of the target time window can be used as the second part of the time window.

[0268] Step S2224a: Select the samples whose generation time corresponds to the first part of the time window as the training set, and select the samples whose generation time corresponds to the second part of the time window as the validation set.

[0269] For example, if the time split ratio is 6:1, the target time window is 7 months long, and the sample set includes samples generated from January to August, then if the target time window corresponds to January to July, then the first part of the time window corresponds to January to June, and the second part of the time window corresponds to July. The samples generated from January to June can be used as the training set, and the samples generated from July can be used as the validation set.

[0270] The embodiments of the present invention split the sample set by using a target time window, which can ensure that the generation time of the samples in the validation set is after the generation time of the samples in the training set. In this way, the problem of data skipping can be avoided by using the later-generated samples for machine learning training and the earlier-generated samples for validating the trained machine learning model.

[0271] Furthermore, splitting the training and validation sets by the beginning and end of the target time window ensures that the generation times of samples in the validation set are adjacent to those in the training set. Optimal hyperparameter values ​​selected based on temporally adjacent training and validation sets are more stable than those selected based on sets with larger time spans. This leads to a more stable machine learning model corresponding to the optimized hyperparameter values, further improving model performance and effectiveness.

[0272] In another embodiment of the present invention, obtaining the splitting rule corresponding to the judgment result and splitting the sample set into a training set and a validation set according to the splitting rule may include the following steps S2221b~S2223b:

[0273] Step S2221b: If the labels in the sample set do not have a temporal relationship, obtain the preset quantity splitting ratio.

[0274] Specifically, the time split ratio can be obtained from the splitting rules. With a splitting rule of 8:1:1, the sample set can be split into one training set and two validation sets, with a split ratio of 8:1:1. With a splitting rule of 9:1, the sample set can be split into one training set and one validation set, with a split ratio of 9:1. With a splitting rule of 7:1:1:1, the sample set can be split into one training set and three validation sets, with a split ratio of 7:1:1:1.

[0275] Step S2222b: Based on the quantity splitting ratio and the number of samples in the sample set, obtain the first quantity for determining the training set and the second quantity for determining the validation set.

[0276] The ratio of the first quantity to the second quantity can conform to the data splitting ratio.

[0277] For example, if the sample set contains 1000 samples, and the partition ratio is 8:1:1, then the first number could be 800, and the second number could be 100 and 100 respectively. If the partition ratio is 9:1, then the first number could be 900, and the second number could be 100. If the partition ratio is 7:1:1:1, then the first number could be 700, and the second number could be 100, 100, and 100 respectively.

[0278] Step S2223b: Select a first number of samples from the sample set to obtain the training set; select a second number of samples to obtain the validation set.

[0279] In an embodiment of the present invention, the sample set may be split into a training set containing a first number of samples and a validation set containing a second number of samples. The samples contained in the validation set and the training set do not overlap, that is, the validation set and the training set do not contain the same samples.

[0280] With 1000 samples in the sample set, if the first number is 800, and the second number is 100 and 100 respectively, the 1000 samples can be split into 800, 100, and 100 samples, serving as the training set, validation set, and validation set, respectively. If the first number is 900, and the second number is 100, the 1000 samples can be split into 900 and 100 samples, serving as the training set, validation set, and validation set, respectively. If the first number is 700, and the second number is 100, 100, and 100 respectively, the 1000 samples can be split into 700, 100, 100, and 100 samples, serving as the training set, validation set, validation set, and validation set, respectively.

[0281] In one embodiment of the present invention, for example in a classification scenario, the samples in the sample set can be classified in advance according to the labels. For each category of samples, they are split according to the quantity splitting ratio to obtain the training set samples and validation set samples of the corresponding category. Then, the training set samples of all categories are merged and the validation set samples of all categories are merged to obtain the training set and validation set.

[0282] The method of splitting samples of each category according to the quantity splitting ratio can refer to the aforementioned steps S2221b to S2223b, and will not be repeated here.

[0283] For example, if the sample data in the sample set consists of images, and the samples in the sample set are divided into three categories based on labels: cats, dogs, and snakes, with sample sizes of 500, 400, and 100 respectively, and the split ratio is 9:1, then the cat samples can be split into 450 and 50, the dog samples into 360 and 40, and the snake samples into 90 and 10. Then, the 450 cat samples, 360 dog samples, and 90 snake samples can be used as the training set, and the 50 cat samples, 40 dog samples, and 10 snake samples can be used as the validation set.

[0284] This embodiment ensures that the proportion of samples of each category is the same in both the training and validation sets. This makes the optimal hyperparameter values ​​obtained from the training and validation sets more stable, and the machine learning model corresponding to the optimal hyperparameter values ​​more stable, thereby further improving the performance and effectiveness of the model.

[0285] In one embodiment of the present invention, the number of validation sets obtained by splitting can be multiple. Then, validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation sets to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation sets can include:

[0286] For each validation set, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the corresponding validation set.

[0287] Selecting optimal hyperparameter values ​​based on the verification score can include:

[0288] For each set of candidate hyperparameter values, determine the comprehensive validation score of the corresponding machine learning model on all validation sets; select the preferred hyperparameter values ​​based on the comprehensive validation score.

[0289] The comprehensive verification score includes any one of the following: mean, median, minimum, and maximum. The mean can be either the geometric mean or the arithmetic mean.

[0290] In this embodiment, for each set of candidate hyperparameter values, the average, median, minimum, or maximum value of the validation scores of the corresponding machine learning model on all validation sets can be determined as the comprehensive validation score of the corresponding candidate hyperparameter values.

[0291] The method of selecting the optimal hyperparameter value based on the comprehensive verification score can be referred to as the method of selecting the optimal hyperparameter value based on the verification score mentioned above, and will not be repeated here.

[0292] <Method Example 4>

[0293] Based on the aforementioned method embodiment one, the method may further include, for example: Figure 3 Steps S3100 to S3500 shown:

[0294] Step S3100: Check whether the event to initiate self-learning has occurred.

[0295] In one embodiment of the present invention, the event may include at least one of the following:

[0296] The preset startup cycle has been reached;

[0297] The preset startup time has been reached;

[0298] The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set is completed.

[0299] Received the specified operation executed by the user.

[0300] In embodiments where the event triggering self-learning includes reaching a preset start-up period, the start-up period can be pre-defined based on the application scenario or specific requirements. For example, the start-up period could be one week, meaning the event triggering self-learning could occur every week.

[0301] In embodiments where the event triggering self-learning includes reaching a preset start time, this start time can be pre-set according to the application scenario or specific needs. For example, the start time could be 8:00 AM on January 1, 2020. In this case, the event triggering self-learning is activated when this time is reached.

[0302] In an embodiment where the self-learning initiation event includes the completion of the step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set, the user may have pre-set a self-learning instruction. In this case, if step S2400 is completed, it is determined that the self-learning initiation event has occurred.

[0303] In embodiments where the event triggering self-learning includes receiving a specified operation from the user, the user may determine, based on the application scenario or specific needs, that self-learning of the machine learning model is required, and then execute the specified operation. This specified operation may be pre-defined according to the application scenario or specific needs. For example, the specified operation may be clicking a preset "Start Self-Learning" button.

[0304] Step S3200: In the event of the event, obtain a new training set and a new validation set.

[0305] Based on the temporal relationship between the labels of the sample set in the aforementioned Embodiment 3, the method for obtaining a new training set and a new validation set may include the following steps S3210 to S3230:

[0306] Step S3210: Obtain the preset sliding duration.

[0307] The sliding duration can be preset according to the application scenario or specific needs. For example, the sliding duration can be one month.

[0308] Step S3220: Slide the target time window according to the sliding duration.

[0309] Specifically, this can be achieved by sliding a target time window on the timeline, with the sliding distance of the target time window being the sliding duration.

[0310] Step S3230: Select the samples corresponding to the generation time and the first part of the sliding time window as the new training set, and select the samples corresponding to the generation time and the second part of the sliding time window as the new validation set.

[0311] For example, if the sample set contains samples generated from January to September, the target time window is 7 months long, the time split ratio is 6:1, the training set obtained in step S2200 consists of samples generated from January to June, the validation set consists of samples generated in July, and the sliding duration can be one month, then after sliding the target time window according to the sliding duration, the target time window corresponds to the period from February to August. Specifically, the first part of the sliding time window corresponds to the period from February to July, and the second part of the sliding time window corresponds to the period in August.

[0312] Correspondingly, one could select the samples generated from the first part of the sliding time window—from February to July—as the new training set, and select the samples generated from the second part of the sliding time window—from August—as the new validation set.

[0313] In this embodiment, a new training set and a new validation set for self-learning are selected by moving the target time window, so that the selected validation set differs from each other by a sliding time period. This ensures the stability of the optimal hyperparameter values ​​selected through self-learning, making the machine learning model corresponding to the optimal hyperparameter values ​​more stable, and further improving the model's performance and effectiveness.

[0314] Each validation set spans a month, ensuring the stability of the results while using a sliding time window method to generate multiple training and validation sets based on a single sample set, allowing the model to freely call upon these sets during the self-learning process.

[0315] In another embodiment of the present invention, the labels of the sample set do not have a temporal relationship. Therefore, obtaining a new training set and a new validation set may include: obtaining a new sample set and splitting the new sample set into a new training set and a new validation set according to a preset splitting rule.

[0316] Step S3300: Iteratively train the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new training set.

[0317] Step S3400: Validate the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new validation set, and obtain the new validation score of the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​on the new validation set.

[0318] In one embodiment of the present invention, the candidate hyperparameter values ​​used in step S3400 may be obtained through step S2300.

[0319] In another embodiment of the present invention, the candidate hyperparameter values ​​used in step S3400 may be candidate hyperparameter values ​​obtained by re-searching for hyperparameters according to a preset hyperparameter tuning algorithm and training a machine learning model based on a new training set.

[0320] Step S3500: Select the preferred hyperparameter value from multiple candidate hyperparameter values ​​based on the new verification score.

[0321] In one embodiment of the present invention, the candidate hyperparameter values ​​used in step S3400 are obtained through step S2300. Therefore, for each group of candidate hyperparameter values, the preferred hyperparameter values ​​can be selected based on the comprehensive validation score of the corresponding machine learning model on all validation sets (wherein, the comprehensive validation score can be determined based on the validation score obtained in step S2400 and the new validation score obtained in step S3400, for example, the average, median, minimum, or maximum value of these validation scores).

[0322] In another embodiment of the present invention, the candidate hyperparameter values ​​used in step S3400 may be obtained by re-searching for hyperparameters according to a preset hyperparameter tuning algorithm and training the machine learning model based on a new training set. In this case, preferred hyperparameter values ​​may be selected from the candidate hyperparameter values ​​used in step S3400 based solely on the new verification score.

[0323] In the embodiments of the present invention, the selection of preferred hyperparameter values ​​through self-learning can ensure the generalization effect of the preferred hyperparameter values ​​on different training and validation sets, further improve the stability of the preferred hyperparameter values, and thus improve the performance and effect of the machine learning model corresponding to the trained preferred hyperparameter values.

[0324] <Example 1>

[0325] For example, in anti-fraud applications, randomly dividing the training and validation sets may bias the results. Specifically, the main impacts are as follows: due to random division, samples of users caught later (e.g., reported in September 2019) may appear in the training set earlier, while earlier samples (e.g., records from August 2019) may appear in the validation set. This means the model already knows in advance that the person will be caught (acting as a prophet). As a result, the model may perform better than the real level on the validation set, but its effectiveness may drop sharply in practical applications.

[0326] The following describes the method for implementing automatic hyperparameter selection, taking the example of a sample set with temporal relationships between labels.

[0327] First, the sample set is split into a training set and a validation set according to the splitting rules.

[0328] The splitting rule represents a range division. For example, in an 8:1:1 splitting rule, the '1' represents a basic interval unit, such as a month, a quarter, or even a non-integer time period like 1.5 months (when the total time span is 15 months). Considering timeliness, for training sets with sequentially related tags, the selection of the validation set is generally based on shifting the training set by one basic interval unit. The maximum number of batches of training and validation sets that can be generated can be determined by counting the number of '1's.

[0329] For example, such as Figure 4 As shown, the samples in the sample set were generated from January 2018 to September 2018. With a partitioning rule of 6:1:1:1, the target time window was set to a duration of 7 months, with the first part of the time window lasting 6 months and the second part lasting 1 month. By moving the target time window (sliding for 1 month), a maximum of 3 training and validation sets can be obtained: samples generated from January 2018 to June 2018 are used as training set 1, and samples generated in July 2018 are used as validation set 1; samples generated from February 2018 to July 2018 are used as training set 2, and samples generated in August 2018 are used as validation set 2; samples generated from March 2018 to August 2018 are used as training set 3, and samples generated in September 2018 are used as validation set 3.

[0330] This way, each validation set spans a month, ensuring the stability of the results while using a sliding time window method to generate multiple training and validation sets based on a single sample set, allowing the model to freely call upon these sets during the self-learning process.

[0331] Specifically, the model could return training set 1 and validation set 1 on the first call, training set 2 and validation set 2 on the second call, and then return the corresponding training set and validation set in chronological order.

[0332] The number of validation sets depends on the observation granularity of the validation sets, which can be set by the user based on the actual situation and their own needs. For example, if the model is deployed daily and the user wants to see the stability of the model on different days, the observation granularity is set to days, then September 2018 can be considered to have 30 validation sets; if the user wants to see the stability of the model's prediction performance monthly, the observation granularity is set to months, and there is only 1 validation set.

[0333] For a set of candidate hyperparameter values ​​K1 obtained by performing hyperparameter search according to the preset hyperparameter tuning algorithm, the machine learning model corresponding to the candidate hyperparameter value K1 is trained based on training set 1, and the machine learning model corresponding to the trained candidate hyperparameter value K1 is validated based on validation set 1. The validation score of the machine learning model corresponding to the trained candidate hyperparameter value K1 on validation set 1 is Score1.

[0334] If the self-learning initiation event occurs, a new training set 2 and a new validation set 2 are obtained. The machine learning model corresponding to the candidate hyperparameter value K1 is trained based on the new training set 2, and then validated on the new validation set 2. The validation score of the trained machine learning model corresponding to the candidate hyperparameter value K1 on the new validation set 2 is obtained as Score2. Then, a new training set 3 and a new validation set 3 are obtained. The machine learning model corresponding to the candidate hyperparameter value K1 is trained based on the new training set 3, and then validated on the new validation set 3. The validation score of the trained machine learning model corresponding to the candidate hyperparameter value K1 on the new validation set 3 is obtained as Score3.

[0335] Determine the mean, median, minimum, or maximum of the validation scores Score1, Score2, and Score3 corresponding to the candidate hyperparameter value K1, and obtain the comprehensive validation score of the machine learning model corresponding to the candidate hyperparameter value K1 on all validation sets.

[0336] For each set of candidate hyperparameter values, the comprehensive validation score of the corresponding machine learning model on all validation sets is determined using the above method, and then the preferred hyperparameter values ​​are selected based on the comprehensive validation score.

[0337] <Example 2>

[0338] The following describes the method for implementing automatic hyperparameter selection, taking the example that the labels in the sample set do not have a temporal relationship.

[0339] First, the sample set is split into a training set and a validation set according to the splitting rules.

[0340] The splitting rule represents a quantity division. For example, in the splitting rule 7:1:1:1, 1 represents a basic quantity unit. The maximum number of validation sets that can be generated can be determined by counting the number of 1s.

[0341] For example, by splitting the sample set according to the splitting rule 6:1:1:1:1, a maximum of 1 training set and 4 validation sets can be obtained. Specifically, 60% of the samples in the sample set can be selected as the training set, and the remaining 10%, 10%, 10%, and 10% can be used as validation set 1, validation set 2, validation set 3, and validation set 4, respectively.

[0342] For the set of candidate hyperparameter values ​​K1 obtained by hyperparameter search according to the preset hyperparameter tuning algorithm, the machine learning model corresponding to the candidate hyperparameter value K1 is trained on the training set 1, and the machine learning model corresponding to the candidate hyperparameter value K1 is validated on the validation set 1 and validation set 2 respectively. The validation scores of the machine learning model corresponding to the candidate hyperparameter value K1 are Score1 on the validation set 1, Score2 on the validation set 2, Score3 on the validation set 3, and Score4 on the validation set 4.

[0343] Determine the mean, median, minimum, or maximum of the validation scores Score1, Score2, Score3, and Score4 corresponding to the candidate hyperparameter value K1, and obtain the comprehensive validation score of the machine learning model corresponding to the candidate hyperparameter value K1 on all validation sets.

[0344] For each set of candidate hyperparameter values, the comprehensive validation score of the corresponding machine learning model on all validation sets is determined using the above method, and then the preferred hyperparameter values ​​are selected based on the comprehensive validation score.

[0345] <Device Embodiment>

[0346] In this embodiment, a device 5000 for implementing automatic hyperparameter selection is provided, such as... Figure 5As shown, the system includes a sample acquisition module 5100, a sample splitting module 5200, a hyperparameter tuning search module 5300, a score determination module 5400, and a hyperparameter selection module 5500. The sample acquisition module 5100 acquires a sample set, where each sample in the sample set includes corresponding data and a label. The sample splitting module 5200 splits the sample set into a training set and a validation set according to a preset splitting rule. The hyperparameter tuning search module 5300 searches for hyperparameters according to a preset hyperparameter tuning algorithm and trains a machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values. The score determination module 5400 validates the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set to obtain a validation score for the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set. The hyperparameter selection module 5500 selects at least one set of preferred hyperparameter values ​​from the multiple sets of candidate hyperparameter values ​​based on the validation score.

[0347] In one embodiment of the present invention, the sample splitting module 5200 can be used for:

[0348] Determine whether the labels in the sample set have a temporal relationship;

[0349] Obtain the splitting rules corresponding to the judgment results, and split the sample set into training set and validation set according to the splitting rules.

[0350] In one embodiment of the present invention, obtaining the splitting rule corresponding to the judgment result and splitting the sample set into a training set and a validation set according to the splitting rule includes:

[0351] Given that the labels in the sample set have a temporal relationship, obtain the time split ratio of the training set and the validation set;

[0352] Select the target time window with a set duration;

[0353] The target time window is divided into a first part of the time window for selecting the training set and a second part of the time window for selecting the validation set, based on the time splitting ratio.

[0354] Samples whose generation time corresponds to the first part of the time window are selected as the training set, and samples whose generation time corresponds to the second part of the time window are selected as the validation set.

[0355] In one embodiment of the present invention, the apparatus 5000 for implementing automatic hyperparameter selection may further include:

[0356] The self-learning detection module is used to detect whether the event that initiates self-learning has occurred;

[0357] A new sample acquisition module is used to acquire new training sets and new validation sets in the event of an event.

[0358] The iterative training module is used to iteratively train the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new training set.

[0359] The new score determination module is used to validate the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new validation set, and obtain the new validation score of the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​on the new validation set.

[0360] A new hyperparameter selection module is used to reselect preferred hyperparameter values ​​from multiple candidate hyperparameter values ​​based on the new validation score.

[0361] In one embodiment of the present invention, obtaining a new training set and a new validation set includes:

[0362] Get the preset sliding duration;

[0363] Slide the target time window according to the sliding duration;

[0364] The samples corresponding to the generation time and the first part of the sliding time window are selected as the new training set, and the samples corresponding to the generation time and the second part of the sliding time window are selected as the new validation set.

[0365] In one embodiment of the present invention, the event may include at least one of the following:

[0366] The preset startup cycle has been reached;

[0367] The preset startup time has been reached;

[0368] The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set is completed.

[0369] Received the specified operation executed by the user.

[0370] In one embodiment of the present invention, obtaining the splitting rule corresponding to the judgment result and splitting the sample set into a training set and a validation set according to the splitting rule includes:

[0371] When the labels in the sample set do not have a temporal relationship, obtain the preset quantity splitting ratio;

[0372] Based on the quantity splitting ratio and the number of samples in the sample set, we obtain the first quantity used to determine the training set and the second quantity used to determine the validation set.

[0373] The first number of samples are selected from the sample set to obtain the training set; the second number of samples are selected to obtain the validation set.

[0374] In one embodiment of the present invention, determining whether the labels of the sample set have a temporal relationship includes:

[0375] Identify application scenarios for machine learning models;

[0376] Given a specific application scenario, determine that the labels in the sample set have a temporal relationship;

[0377] or,

[0378] To determine the type of data content in the sample set;

[0379] Given that the data content is of a specified type, the labels in the sample set are determined to have a temporal relationship.

[0380] In one embodiment of the present invention, when the labels of samples in the sample set are continuous values, the apparatus 5000 for automatic hyperparameter selection may further include:

[0381] This module is used to discretize the labels.

[0382] In one embodiment of the present invention, the parameter tuning search module is used for:

[0383] Multiple rounds of training are conducted. In each round of training, a set of one or more initial hyperparameter values ​​are obtained by searching for hyperparameters according to the hyperparameter tuning algorithm. One or more corresponding machine learning models are trained based on the training set and the set of one or more initial hyperparameter values, and the corresponding model scores are obtained.

[0384] Based on the model scores obtained from multiple rounds of training, multiple sets of candidate hyperparameter values ​​are selected.

[0385] In one embodiment of the present invention, the selection of multiple sets of candidate hyperparameter values ​​based on model scores obtained from multiple rounds of training includes:

[0386] Select initial hyperparameter values ​​whose training scores exceed a preset first score threshold as candidate hyperparameter values; or,

[0387] The initial hyperparameter values ​​are sorted in descending order based on the training scores, and the first sorted value of each initial hyperparameter value is obtained.

[0388] Select an initial hyperparameter value within a first set range as a candidate hyperparameter value.

[0389] In one embodiment of the present invention, the apparatus 5000 for implementing automatic hyperparameter selection may further include:

[0390] This module is used to interrupt the search for multiple sets of initial hyperparameter values ​​for a machine learning model in response to a request to interrupt the search for hyperparameter values.

[0391] This module is used to continue searching for multiple sets of initial hyperparameter values ​​for a machine learning model in response to a request to continue searching for hyperparameter values.

[0392] In one embodiment of the present invention, the parameter tuning algorithm is set in a preset infinite loop program.

[0393] The device 5000 for automatic hyperparameter selection may also include:

[0394] This module is used to save the initial hyperparameter values ​​obtained each time by the hyperparameter tuning algorithm to the history.

[0395] In response to the request to continue searching for hyperparameter values, the search continues for multiple sets of initial hyperparameter values ​​for the machine learning model, including:

[0396] In response to a request to continue searching for hyperparameter values, load the history and continue searching for the next set of initial hyperparameter values ​​for the machine learning model based on the history.

[0397] In one embodiment of the present invention, the number of verification sets is multiple, and the score determination module 5400 can be used for:

[0398] For each validation set, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the corresponding validation set.

[0399] Based on the verification score, the preferred hyperparameter values ​​include:

[0400] For each set of candidate hyperparameter values, determine the comprehensive validation score of the corresponding machine learning model on all validation sets;

[0401] Select the optimal hyperparameter values ​​based on the comprehensive verification score;

[0402] The comprehensive verification score includes any one of the following: mean, median, minimum, and maximum.

[0403] In one embodiment of the present invention, the hyperparameter selection module 5500 can be used for:

[0404] Candidate hyperparameter values ​​whose verification scores exceed a preset second score threshold are selected as preferred hyperparameter values;

[0405] or,

[0406] The candidate hyperparameter values ​​are sorted in descending order based on the verification scores, and a second sorting value is obtained for each group of candidate hyperparameter values.

[0407] The candidate hyperparameter values ​​within the second set range of the second sorting value are selected as the preferred hyperparameter values.

[0408] In one embodiment of the present invention, the parameter tuning algorithm includes any one or more of the following: grid search algorithm, random search algorithm, and Bayesian search algorithm.

[0409] In one embodiment of the present invention, the apparatus 5000 for implementing automatic hyperparameter selection may further include:

[0410] A module that provides a settings entry point for configuring application scenarios for machine learning models;

[0411] This module is used to obtain the application scenario input through the setting entry point;

[0412] This module is used to obtain a sample set based on the input application scenario.

[0413] In one embodiment of the present invention, the apparatus 5000 for implementing automatic hyperparameter selection may further include:

[0414] This module is used to display preferred hyperparameter values.

[0415] In one embodiment of the present invention, the apparatus 5000 for implementing automatic hyperparameter selection may further include:

[0416] A module used to provide an entry point for inputting splitting rules;

[0417] A module used to obtain splitting rules through the entry point.

[0418] In one embodiment of the present invention, the apparatus 5000 for implementing automatic hyperparameter selection may further include:

[0419] Use the module to identify the application scenarios for which machine learning models are applicable;

[0420] A module used to find application projects that match the application scenario;

[0421] This module is used to input the machine learning model corresponding to the optimized hyperparameter values ​​after training into the application project.

[0422] Those skilled in the art will understand that device 5000 can be implemented in various ways. For example, device 5000 can be implemented by configuring a processor with instructions. For example, instructions can be stored in ROM, and when the device is started, the instructions can be read from ROM into a programmable device to implement device 5000. For example, device 5000 can be embedded in a special-purpose device (e.g., ASIC). Device 5000 can be divided into independent units, or they can be combined together. Device 5000 can be implemented by one of the various implementation methods described above, or by a combination of two or more of the various implementation methods described above.

[0423] In this embodiment, the device 5000 can have various implementation forms. For example, the device 5000 can be any functional module running in a software product or application that provides model training services, or a peripheral embedding, plug-in, patch, etc. of such software products or applications, or it can be the software products or applications themselves.

[0424] <System Implementation>

[0425] In this embodiment, as Figure 6 As shown, a system 6000 is also provided, comprising at least one computing device 6100 and at least one storage device 6200. The at least one storage device 6200 is used to store executable instructions for controlling the at least one computing device 6100 to execute a method for implementing automatic hyperparameter selection according to any embodiment of the present invention.

[0426] In this embodiment, the system 6000 can be a mobile phone, tablet computer, handheld computer, desktop computer, laptop computer, workstation, game console, or a distributed system composed of multiple devices.

[0427] Computer-readable storage media

[0428] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method for automatic hyperparameter selection as described in any embodiment of the present invention.

[0429] This invention can be an apparatus, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0430] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0431] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0432] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0433] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0434] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0435] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0436] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0437] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for implementing automatic hyperparameter selection, applied in anti-fraud scenarios or image classification scenarios, comprising: Provides a settings entry point for configuring application scenarios for machine learning models; Obtain the application scenario input through the settings entry point; Based on the input application scenario, obtain a sample set in the anti-fraud scenario or the image classification scenario. Each sample in the sample set includes corresponding data and labels. In the case of the anti-fraud scenario, the data is the user's transaction data over a period of time, and the labels indicate whether the user is a fraudulent user. In the case of the image classification scenario, the data is image data, and the labels are image categories. The sample set is split into a training set and a validation set according to a preset splitting rule; The hyperparameter search is performed according to the preset hyperparameter tuning algorithm, and the machine learning model is trained based on the training set to obtain multiple sets of candidate hyperparameter values. Based on the validation set, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set. Based on the verification score, at least one set of preferred hyperparameter values ​​is selected from the plurality of candidate hyperparameter values; The preferred hyperparameter values ​​are shown. The step of splitting the sample set into a training set and a validation set according to a preset splitting rule includes: Obtain the application scenarios of the machine learning model; In the case where the application scenario is the anti-fraud scenario, it is determined that the labels in the sample set have a temporal relationship; Obtain the splitting rules corresponding to the judgment results, and split the sample set into a training set and a validation set according to the splitting rules. The method further includes: Find application projects that match the anti-fraud scenario or the image classification scenario; The machine learning model corresponding to the selected hyperparameter values ​​after training is input into the application project.

2. The method according to claim 1, wherein obtaining the splitting rule corresponding to the judgment result and splitting the sample set into a training set and a validation set according to the splitting rule comprises: Given that the labels in the sample set have a temporal relationship, obtain the time split ratio of the training set and the validation set; Select the target time window with a set duration; The target time window is divided into a first part of the time window for selecting the training set and a second part of the time window for selecting the validation set according to the time split ratio. The samples whose generation time corresponds to the first part of the time window are selected as the training set, and the samples whose generation time corresponds to the second part of the time window are selected as the validation set.

3. The method according to claim 2, further comprising: Detect whether the event that initiates self-learning has occurred; In the event that the event occurs, a new training set and a new validation set are obtained; The machine learning model corresponding to each set of candidate hyperparameter values ​​is iteratively trained based on the new training set. The iterative machine learning model corresponding to each set of candidate hyperparameter values ​​is validated based on the new validation set to obtain a new validation score for the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​on the new validation set. Based on the new verification score, preferred hyperparameter values ​​are reselected from the multiple sets of candidate hyperparameter values.

4. The method according to claim 3, wherein obtaining the new training set and the new validation set comprises: Get the preset sliding duration; The target time window is slid according to the specified sliding duration; The samples corresponding to the generation time and the first part of the sliding time window are selected as the new training set, and the samples corresponding to the generation time and the second part of the sliding time window are selected as the new validation set.

5. The method of claim 3, wherein the event comprises at least one of the following: The preset startup cycle has been reached; The preset startup time has been reached; The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set has been completed. Received the specified operation executed by the user.

6. The method according to claim 1, wherein when the labels of the samples in the sample set are continuous values, the method further comprises: The labels are discretized.

7. The method according to claim 1, wherein the step of searching for hyperparameters according to a preset hyperparameter tuning algorithm and training a machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values ​​includes: Multiple rounds of training are performed, wherein in each round of training, one or more sets of initial hyperparameter values ​​are obtained by hyperparameter search according to the hyperparameter tuning algorithm, one or more corresponding machine learning models are trained based on the training set and the one or more sets of initial hyperparameter values, and the corresponding model scores are obtained. Based on the model scores obtained from the multiple rounds of training, multiple sets of candidate hyperparameter values ​​are selected.

8. The method according to claim 7, wherein selecting multiple sets of candidate hyperparameter values ​​based on the model scores obtained in the multiple rounds of training includes: An initial hyperparameter value that exceeds a preset first score threshold is selected as the candidate hyperparameter value; or, The initial hyperparameter values ​​are sorted in descending order based on the training scores, and the first sorting value of each initial hyperparameter value is obtained. The initial hyperparameter value within a first set range of the first sorting value is selected as the candidate hyperparameter value.

9. The method according to claim 7, further comprising: In response to a request to interrupt the search for hyperparameter values, the search for multiple sets of initial hyperparameter values ​​for the machine learning model is interrupted. In response to the request to continue searching for hyperparameter values, the search continues for multiple sets of initial hyperparameter values ​​for the machine learning model.

10. The method according to claim 9, wherein the parameter tuning algorithm is set in a preset infinite loop program. The method further includes: The initial hyperparameter values ​​obtained each time according to the parameter tuning algorithm are saved to the history. In response to a request to continue searching for hyperparameter values, continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model includes: In response to a request to continue searching for hyperparameter values, the history is loaded, and the search for the next set of initial hyperparameter values ​​for the machine learning model is continued based on the history.

11. The method according to claim 1, wherein the number of verification sets is multiple. The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set, and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set, includes: For each of the aforementioned validation sets, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the corresponding validation set. The step of selecting preferred hyperparameter values ​​based on the verification score includes: For each set of candidate hyperparameter values, determine the comprehensive validation score of the corresponding machine learning model on all validation sets; Select the optimal hyperparameter values ​​based on the comprehensive verification score; The comprehensive verification score includes any one of the following: average, median, minimum, and maximum.

12. The method according to claim 1, wherein selecting at least one set of preferred hyperparameter values ​​from the plurality of candidate hyperparameter values ​​based on the verification score comprises: Candidate hyperparameter values ​​whose verification scores exceed a preset second score threshold are selected as the preferred hyperparameter values; or, The candidate hyperparameter values ​​are sorted in descending order based on the verification scores, and a second sorting value is obtained for each group of candidate hyperparameter values. The candidate hyperparameter values ​​within a second set range of the second sorting value are selected as the preferred hyperparameter values.

13. The method according to claim 1, wherein the parameter tuning algorithm includes any one or more of the following: grid search algorithm, random search algorithm, and Bayesian search algorithm.

14. The method according to any one of claims 1 to 13, further comprising: Provide an entry point for inputting the splitting rules; The splitting rules are obtained through the aforementioned entry point.

15. An apparatus for implementing automatic hyperparameter selection, applied in anti-fraud scenarios or image classification scenarios, comprising: The sample acquisition module is used to acquire a sample set in the anti-fraud scenario or the image classification scenario. Each sample in the sample set includes corresponding data and a label. The data is the user's transaction data over a period of time, and the label indicates whether the user is a fraudulent user. In the case of the image classification scenario, the data is image data, and the label is the image category. The sample splitting module is used to split the sample set into a training set and a validation set according to a preset splitting rule; The hyperparameter tuning search module is used to search for hyperparameters according to a preset hyperparameter tuning algorithm and to train a machine learning model based on the training set to obtain multiple sets of candidate hyperparameter values. The scoring determination module is used to verify the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the verification set, and to obtain the verification score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the verification set. The hyperparameter selection module is used to select at least one set of preferred hyperparameter values ​​from the plurality of candidate hyperparameter values ​​based on the verification score. Specifically, the sample splitting module is used for: Obtain the application scenarios of the machine learning model; In the case where the application scenario is the anti-fraud scenario, it is determined whether the labels of the sample set have a temporal relationship. Obtain the splitting rules corresponding to the judgment results, and split the sample set into a training set and a validation set according to the splitting rules. The device further includes: A module for finding application projects that match the anti-fraud scenario or the image classification scenario; Module for inputting the machine learning model corresponding to the trained preferred hyperparameter values ​​into the application project; A module for providing a settings entry point for configuring the application scenarios of the machine learning model; A module for obtaining the application scenario input through the setting entry; A module for obtaining the sample set based on the application scenario input; Module used to display the preferred hyperparameter values.

16. The apparatus according to claim 15, wherein obtaining the splitting rule corresponding to the judgment result and splitting the sample set into a training set and a validation set according to the splitting rule comprises: Given that the labels in the sample set have a temporal relationship, obtain the time split ratio of the training set and the validation set; Select the target time window with a set duration; The target time window is divided into a first part of the time window for selecting the training set and a second part of the time window for selecting the validation set according to the time split ratio. The samples whose generation time corresponds to the first part of the time window are selected as the training set, and the samples whose generation time corresponds to the second part of the time window are selected as the validation set.

17. The apparatus of claim 16, further comprising: The self-learning detection module is used to detect whether the event that initiates self-learning has occurred; A new sample acquisition module is used to acquire a new training set and a new validation set in the event said event occurs; The iterative training module is used to iteratively train the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new training set. The new score determination module is used to verify the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​based on the new validation set, and to obtain a new validation score of the iterative machine learning model corresponding to each set of candidate hyperparameter values ​​on the new validation set. A new hyperparameter selection module is used to reselect preferred hyperparameter values ​​from the multiple sets of candidate hyperparameter values ​​based on the new verification score.

18. The apparatus of claim 17, wherein acquiring the new training set and the new validation set comprises: Get the preset sliding duration; The target time window is slid according to the specified sliding duration; The samples corresponding to the generation time and the first part of the sliding time window are selected as the new training set, and the samples corresponding to the generation time and the second part of the sliding time window are selected as the new validation set.

19. The apparatus of claim 17, wherein the event includes at least one of the following: The preset startup cycle has been reached; The preset startup time has been reached; The step of validating the machine learning model corresponding to each set of candidate hyperparameter values ​​based on the validation set and obtaining the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the validation set has been completed. Received the specified operation executed by the user.

20. The apparatus of claim 15, wherein when the labels of the samples in the sample set are continuous values, the apparatus further comprises: A module for discretizing the labels.

21. The apparatus according to claim 15, wherein the parameter tuning search module is used for: Multiple rounds of training were conducted, among which, In each round of training, a set of one or more sets of initial hyperparameter values ​​are obtained by hyperparameter search according to the hyperparameter tuning algorithm. One or more corresponding machine learning models are trained based on the training set and the set of one or more sets of initial hyperparameter values, and the corresponding model scores are obtained. Based on the model scores obtained from the multiple rounds of training, multiple sets of candidate hyperparameter values ​​are selected.

22. The apparatus according to claim 21, wherein selecting multiple sets of candidate hyperparameter values ​​based on the model scores obtained in the multiple rounds of training includes: An initial hyperparameter value that exceeds a preset first score threshold is selected as the candidate hyperparameter value; or, The initial hyperparameter values ​​are sorted in descending order based on the training scores, and the first sorting value of each initial hyperparameter value is obtained. The initial hyperparameter value within a first set range of the first sorting value is selected as the candidate hyperparameter value.

23. The apparatus of claim 15, further comprising: A module for interrupting the search for multiple sets of initial hyperparameter values ​​of the machine learning model in response to a request to interrupt the search for hyperparameter values; A module for continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model in response to a request to continue searching for hyperparameter values.

24. The apparatus according to claim 23, wherein the parameter tuning algorithm is set in a preset infinite loop program. The device further includes: This module is used to save the initial hyperparameter values ​​obtained each time according to the parameter tuning algorithm to the history. In response to a request to continue searching for hyperparameter values, continuing to search for multiple sets of initial hyperparameter values ​​for the machine learning model includes: In response to a request to continue searching for hyperparameter values, the history is loaded, and the search for the next set of initial hyperparameter values ​​for the machine learning model is continued based on the history.

25. The apparatus according to claim 15, wherein the number of the verification sets is plurality. The score determination module is used for: For each of the aforementioned validation sets, the machine learning model corresponding to each set of candidate hyperparameter values ​​is validated to obtain the validation score of the machine learning model corresponding to each set of candidate hyperparameter values ​​on the corresponding validation set. The step of selecting preferred hyperparameter values ​​based on the verification score includes: For each set of candidate hyperparameter values, determine the comprehensive validation score of the corresponding machine learning model on all validation sets; Select the optimal hyperparameter values ​​based on the comprehensive verification score; The comprehensive verification score includes any one of the following: average, median, minimum, and maximum.

26. The apparatus according to claim 15, wherein the hyperparameter selection module is used for: Candidate hyperparameter values ​​whose verification scores exceed a preset second score threshold are selected as the preferred hyperparameter values; or, The candidate hyperparameter values ​​are sorted in descending order based on the verification scores, and a second sorting value is obtained for each group of candidate hyperparameter values. The candidate hyperparameter values ​​within a second set range of the second sorting value are selected as the preferred hyperparameter values.

27. The apparatus according to claim 15, wherein the parameter tuning algorithm includes any one or more of the following: grid search algorithm, random search algorithm, and Bayesian search algorithm.

28. The apparatus according to any one of claims 15 to 27, further comprising: A module for providing an entry point for inputting the splitting rules; A module used to obtain the splitting rules through the entry point.

29. A system comprising at least one computing device and at least one storage device, wherein, The at least one storage device is used to store instructions for controlling the at least one computing device to perform the method according to any one of claims 1 to 14.

30. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method as described in any one of claims 1 to 14 when executed by a processor.

Images