Hyper-parameter optimization system and method

By introducing hyperparameter multiplexers and adjusters into the hyperparameter optimization system, combining hardware specifications and performance limitations, hyperparameter search is optimized, and high-dimensional search problems are solved, achieving efficient and real-time performance of the model on edge devices.

CN120542593APending Publication Date: 2025-08-26DELTA ELECTRONICS INC(CN)
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410208631.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-02-26
Publication Date
2025-08-26

AI Technical Summary

Technical Problem

The existing hyperparameter optimization method is difficult to converge to the optimal solution in the high-dimensional search space, and fails to effectively combine the search of model-related and non-related hyperparameters, resulting in insufficient model inference speed on edge devices and difficult to meet real-time requirements.

Method used

By introducing a hyperparameter multiplexer, combining hardware specifications and performance limitations, the model-related hyperparameter combination is selected from the hyperparameter set, and the model is searched through the hyperparameter adjuster to establish a machine learning model.

Benefits of technology

Reduce the search dimension, improve the search efficiency, ensure that the hyperparameter combination can meet the model performance and inference speed requirements under hardware limitations, and optimize the model architecture and input resolution to meet the real-time requirements of edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120542593A_ABST
    Figure CN120542593A_ABST
Patent Text Reader

Abstract

The invention provides a hyper-parameter optimization system and a hyper-parameter optimization method, which comprise the following steps of: selecting a model-related hyper-parameter combination which conforms to hardware specification limitation and efficiency limitation from a hyper-parameter set by searching a model database; based on the model related hyper-parameter combination, searching the optimal hyper-parameter in each group of model non-closed hyper-parameters through a hyper-parameter adjuster to obtain an optimal hyper-parameter combination; and establishing a machine learning model based on the optimal hyper-parameter combination.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to machine learning, and in particular to a hyperparameter optimization system and method. Background Art

[0002] In machine learning, unlike learnable parameters (such as the weights and biases in a neural network, or the depth of a decision tree and the minimum number of samples per node), hyperparameters (such as the model architecture, learning rate, epochs, the weight of the loss function, the average brightness of the input image, etc.) are not automatically adjusted by the training algorithm during the learning process. Instead, they are given before model training to define and control the model learning process. Using different hyperparameters will produce different trained models. Therefore, the setting of hyperparameters has a decisive influence on the performance and generalization of the model.

[0003] Generally speaking, the process of hyperparameter optimization involves first defining an objective function for evaluating the quality of a model and a search space for individual hyperparameters. Then, based on the various hyperparameters in the search space, various prototype models are trained to find the optimal hyperparameters that can train the best prototype model (i.e., the prototype model with the best evaluation result using the objective function). Traditional hyperparameter optimization is often performed manually, and the process requires a lot of computing resources, time, and manpower. Some existing approaches automate hyperparameter optimization through a hyperparameter tuner. More specifically, the hyperparameter tuner determines the hyperparameters to be evaluated in the next iteration based on the evaluation results of the prototype model using the objective function.

[0004] While these automated hyperparameter optimization methods can somewhat save computational resources, time, and manpower compared to manual approaches, they still face significant challenges in practice. First, the dimensionality of the hyperparameter search space is often too high, resulting in inability to converge to the optimal solution, or the search process being extremely slow or even stuck. Second, these approaches fail to search both model-dependent hyperparameters (such as model architecture and input resolution) and model-independent hyperparameters (such as training-related hyperparameters, loss function-related hyperparameters, or data augmentation-related hyperparameters). The combination of the local optimum found in each of these two approaches may not necessarily yield the global optimum. Finally, models are increasingly deployed on edge devices with limited computing power, while still requiring high real-time model inference performance. For example, in autonomous driving scenarios, models must quickly infer traffic sign color and vehicle distance to enable the system to respond appropriately and in real time. However, these approaches fail to account for hardware resource constraints.

[0005] Therefore, a hyperparameter optimization system and method are needed to solve the above problems. Summary of the Invention

[0006] One embodiment of the present invention provides a hyperparameter optimization system comprising a storage unit and a processing unit. The storage unit stores a program, a model database, and a hyperparameter set. The hyperparameter set comprises multiple sets of model-related hyperparameters and multiple sets of model-independent hyperparameters. The processing unit loads the program from the storage device to execute the following steps: selecting a model-related hyperparameter combination from the hyperparameter set that meets hardware specification constraints and performance constraints by searching the model database; based on the model-related hyperparameter combination, searching for the optimal hyperparameter among each set of model-independent hyperparameters using a hyperparameter adjuster to obtain an optimal hyperparameter combination; and establishing a machine learning model based on the optimal hyperparameter combination.

[0007] One embodiment of the present invention provides a hyperparameter optimization method implemented by a computer system, which includes the following steps: by searching a model database, selecting a model-related hyperparameter combination that meets hardware specification constraints and performance constraints from a hyperparameter set; based on the model-related hyperparameter combination, searching for the best hyperparameters in each group of model-independent hyperparameters through a hyperparameter adjuster to obtain an optimal hyperparameter combination; and establishing a machine learning model based on the optimal hyperparameter combination.

[0008] The hyperparameter optimization solution provided by the present disclosure can reduce the required search dimensions by grouping hyperparameters to avoid the curse of dimensionality and improve search efficiency. Moreover, by searching for model-related hyperparameters and model-independent hyperparameters together, it can be ensured that they can converge to the global optimal solution together. In addition, by evaluating the model size and input resolution upper limit supported by the hardware, it can be ensured that the searched hyperparameters can meet the model performance requirements while also allowing the hardware to achieve a suitable inference speed. It can also help find the minimum model architecture and input resolution that can meet the performance requirements under limited hardware conditions, taking into account multiple requirements such as model performance, inference speed, and hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present invention will be better understood from the following description of exemplary embodiments with accompanying drawings. In addition, it should be understood that in the flowcharts of the present invention, the execution order of each block may be changed, and / or certain blocks may be changed, deleted, or merged.

[0010] Figure 1 A software architecture diagram of an existing hyperparameter optimization system is shown.

[0011] Figure 2 4 is a software architecture diagram of a hyperparameter optimization system according to an embodiment of the present disclosure.

[0012] Figure 3 1 is a hardware architecture diagram illustrating a hyperparameter optimization system according to an embodiment of the present disclosure.

[0013] Figure 4 Shows an example of a database schema for a model database.

[0014] Figure 5 is a tree diagram illustrating one exemplary embodiment of a set of hyperparameters.

[0015] Figure 6 4 is a flowchart of a hyperparameter optimization method according to an embodiment of the present disclosure.

[0016] Figure 7 is a flowchart of a hyperparameter optimization method according to another embodiment of the present disclosure.

[0017] Description of reference numerals:

[0018] 10: Hyperparameter Optimization System

[0019] 100: Dataset

[0020] 101: Training Dataset

[0021] 102: Validation Dataset

[0022] 103: Prototype Model

[0023] 104: Verification result

[0024] 105: Hyperparameters

[0025] 110: Hyperparameter Tuner

[0026] 120: Model training procedure

[0027] 130: Model Validation Procedure

[0028] 20: Hyperparameter Optimization System

[0029] 201: Hardware specification limitations

[0030] 202: Performance Limitation

[0031] 203: Model Database

[0032] 204: Hyperparameter Set

[0033] 210: Hyperparameter Multiplexer

[0034] 31: Processing unit

[0035] 32: Storage unit

[0036] 311: Processor

[0037] 312: Random Access Memory

[0038] 305: Procedure

[0039] 401: Model Data Table

[0040] 402: Input resolution data table

[0041] 403: Hardware Datasheet

[0042] 404: Model performance data table

[0043] 51: Model-related hyperparameters

[0044] 52: Model is not related to hyperparameters

[0045] 511: Model Architecture

[0046] 512: Input resolution

[0047] 520: Training related subset

[0048] 521: Learning Rate

[0049] 522: Momentum

[0050] 523: training cycle (epoch)

[0051] 530: Loss function related subset

[0052] 531: Loss Function Form

[0053] 532: Weighted Combination

[0054] 540: Visual attribute related subset

[0055] 541: Color Space

[0056] 542: Average brightness

[0057] 550: Data Enhancement Related Subset

[0058] 551: Rotation angle

[0059] 552: Translation

[0060] 553: Flip Mode

[0061] 554: Scaling

[0062] 600: Hyperparameter Optimization Methods

[0063] S601-S603: Steps

[0064] 700: Hyperparameter Optimization Methods

[0065] S701-S703: Steps DETAILED DESCRIPTION

[0066] The following description lists various embodiments of the present invention, but is not intended to limit the present invention. The actual scope of the invention is defined by the claims.

[0067] In the various embodiments listed below, the same reference numerals will be used to represent the same or similar elements or components.

[0068] In this specification and claims, serial numbers, such as "first", "second", etc., are only for convenience of description and have no sequential relationship with each other.

[0069] In this specification, descriptions of embodiments of an apparatus or system also apply to embodiments of a method, and vice versa.

[0070] Unless otherwise specified, the terms "parameter" or "hyperparameter" in this specification refer to "parameter value" or "hyperparameter value" rather than "parameter form" or "hyperparameter form".

[0071] Figure 1 FIG. 1 shows a software architecture diagram of an existing hyperparameter optimization system 10. Figure 1As shown, the hyperparameter optimization system 10 may involve the operation of a hyperparameter adjuster 110 , a model training program 120 , and a model validation program, which will be described in detail below.

[0072] First, the complete dataset 100 is divided into a training dataset 101 and a validation dataset 102, which are used for model training and validation respectively. The dataset 100 can be collected from real-world scenarios or from open-source datasets such as Pascal VOC or CoCo (Common Objects in Context). The dataset 100 is a collection of labeled data, where the label is metadata used to describe the ground truth. The implementation depends on the task type of the model. For example, in the case of object detection tasks, the label is the location and range information of the target object in the image, usually represented by a bounding box; in the case of object recognition tasks, the label is the category of the object (such as male or female, cat or dog); in the case of regression tasks, the label is the variable being predicted, such as distance, price or income.

[0073] Next, the model training process 120 uses the training dataset 101 to train a prototype model 103 based on the initial hyperparameters 105. The prototype model 103 is only used to evaluate the quality of the hyperparameters 105 and will not be deployed in actual application scenarios. However, the prototype model 103 is typically configured to perform the same task type (e.g., object detection, object recognition, or regression) as the model deployed in the actual application scenario. The training of the prototype model 103, i.e., the model training process 120, may involve calculating the gradients of the loss function and iteratively updating the model parameters, such as the combination of various gradient descent (GD) and backpropagation algorithms, which will not be described in detail here.

[0074] Subsequently, the model validation program 130 uses the validation dataset 102 to validate the trained prototype model 103 and outputs corresponding validation results 104. The validation results 104 can be measured using various common model performance metrics, such as accuracy, precision, recall, F1-score, area under the receiver operating characteristic curve (ROC-AUC), or other custom objective functions.

[0075] Because hyperparameters 105 have a decisive impact on the performance of prototype model 103, validation results 104 not only represent the performance of prototype model 103 but also the quality of hyperparameters 105. Therefore, hyperparameter tuner 110 determines the hyperparameters 105 to be evaluated in the next iteration based on validation results 104. Hyperparameter tuner 110 may be implemented, for example, but not limited to, using grid search, random search, Bayesian optimization, or an evolutionary algorithm.

[0076] The above process will be repeated multiple times until the convergence condition is met, that is, the hyperparameter adjuster 110 determines that the optimal verification result 104 can no longer be achieved by adjusting the hyperparameter 105. In this way, the finally obtained hyperparameter 105 can be regarded as the optimal hyperparameter.

[0077] Figure 1 The hyperparameter optimization system 10 still faces severe challenges in many aspects of practice, which have been described in detail in the [Background Art] section and will not be repeated here.

[0078] Figure 2 is a software architecture diagram of a hyperparameter optimization system 20 according to an embodiment of the present disclosure. Figure 1 The hyperparameter optimization system 10 shown, Figure 2 The illustrated hyperparameter optimization system 20 further involves a hyperparameter multiplexer 210 , which will be described in detail below.

[0079] Generally speaking, hyperparameter multiplexer 210 is configured to arrange or select the hyperparameter combinations to be evaluated in the next iteration based on validation results 104. More specifically, hyperparameter multiplexer 210 incorporates hardware specification constraints 201 and performance constraints 202, and by searching model database 203, selects from hyperparameter set 204 the model-dependent hyperparameters and the groups of model-independent hyperparameters to be evaluated in the next iteration. Hyperparameter optimization system 20 then searches for the best solution within each group of model-independent hyperparameters based on the model-dependent hyperparameters given by hyperparameter multiplexer 210, via hyperparameter adjuster 110. The best solutions within each group are combined to form the best model-independent hyperparameter combination under the given model-dependent hyperparameters. The best model-independent hyperparameter combination and the given model-dependent hyperparameters constitute the best hyperparameter combination. Furthermore, the hyperparameter multiplexer 210 may determine whether to adjust the given model-related hyperparameters based on the verification result 104 corresponding to the optimal hyperparameter combination to update the optimal hyperparameter combination.

[0080] The aforementioned hardware specification constraints 201 and performance constraints 202 are conditions that must be met for model deployment in actual application scenarios. For example, hardware specification constraints 201 can be any one or a combination of the target hardware's memory size, clock frequency, or maximum number of supported channels, where the target hardware is the hardware device on which the model is deployed, such as an edge device, accelerator, central processing unit (CPU), graphics processing unit (GPU), or digital signal processor (DSP). Performance constraints 202 can refer to inference speed requirements and / or power consumption limits in actual application scenarios, such as only accepting a specified latency value of 50, 80, or 100 milliseconds, and / or only accepting a power consumption of 300, 500, or 800 milliwatts per task. Hardware specification constraints 201 and performance constraints 202 can be user-entered based on practical needs, or inferred by other programs or machine learning models, and this disclosure is not limited to these requirements.

[0081] The data set recorded in the model database 203 can describe the correspondence between the model, input resolution, and hardware. In other words, the data recorded in the model database 203 describes the performance of various models on various hardware at various input resolutions. This data can be obtained by the developer of the hyperparameter optimization system 20 through actual measurements, or it can be captured from public websites for machine learning model performance benchmarks such as MLPerf, AI Benchmark, and TensorFlow Model Garden, but this disclosure is not limited to this.

[0082] The hyperparameter set 204 may include multiple sets of hyperparameters, which can be further divided into hyperparameters related to the model itself (i.e., so-called "model-related hyperparameters") and hyperparameters not directly related to the model itself (i.e., so-called "model-independent hyperparameters"). As previously mentioned, the hyperparameter optimization system 20 divides the multiple sets of hyperparameters in the hyperparameter set 204 into model-related hyperparameters and model-independent hyperparameters through the hyperparameter multiplexer 210, and enables the hyperparameter adjuster 110 to jointly search for the two, thereby improving the existing search architecture (e.g., Figure 1 The hyperparameter optimization system 10) is missing. Figure 5 , various implementations of the hyperparameter set 204 are described in more detail.

[0083] Figure 3 FIG2 is a diagram showing the hardware architecture of a hyperparameter optimization system 20 according to an embodiment of the present disclosure. From a hardware perspective, the hyperparameter optimization system 20 can be any computer system with computing capabilities, such as a personal computer (e.g., a desktop computer or a laptop computer) or a server computer, or a mobile device such as a tablet computer or a smartphone, but the present disclosure is not limited thereto. Figure 3 As shown, the hyperparameter optimization system 20 may include a processing unit 31 and a storage unit 32. The processing unit 31 may communicate with the storage unit 32 via various wired or wireless communication interfaces, such as a system bus, Serial ATA (SATA), Serial Attached SCSI (SAS), NVM Express (NVMe), PCI Express (PCIe), Universal Serial Bus (USB) interface, USB Type-C interface, Thunderbolt interface, 5G wireless system, Wi-Fi, and / or combinations thereof.

[0084] The processing unit 31 includes one or more general-purpose or special-purpose processors 311 for executing instructions, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SoC), and / or a combination thereof, but the present disclosure is not limited thereto. The processing unit 31 may also include random-access memory (RAM) 312, such as dynamic random-access memory (SRAM), static random-access memory (SRAM), and / or a combination thereof, but the present disclosure is not limited thereto. It should be noted that since the prototype model 103 needs to be loaded into the random access memory 312 in the model training process 120 and the model verification process 130 , the processing unit 31 should have a sufficiently large random access memory 312 .

[0085] Storage unit 32 may be any device having non-volatile memory (e.g., read-only memory, electrically-erasable programmable read-only memory (EEPROM), flash memory, or non-volatile random access memory (NVRAM)), such as a hard disk drive (HDD) array, a solid-state drive (SSD), or an optical disk, but the present disclosure is not limited thereto. Storage unit 32 may be fully or partially deployed locally (i.e., the device where processing unit 31 is located) or remotely, but the present disclosure is not limited thereto.

[0086] In the embodiment of the present disclosure, storage unit 32 stores one or more programs 305, which include multiple instructions corresponding to the aforementioned hyperparameter multiplexer 210, hyperparameter adjuster 110, model training program 120, and model validation program 130. When processing unit 31 loads program 305 from storage unit 32 into random access memory 312, processor 311 runs program 305, executing the corresponding instructions to implement the hyperparameter optimization method. In addition, storage unit 32 also stores data required to execute the hyperparameter optimization method, as well as data generated during execution.

[0087] In one embodiment, the hyperparameter optimization system 20 may further include an input unit, although it is not shown in FIG. Figure 3 The input unit can be any device capable of receiving external input data, such as a mouse, keyboard, keypad, console, touch panel, or voice input device, but this disclosure is not limited thereto. The input unit translates the parameters or information input by the user, such as the aforementioned hardware specification limit 201 and performance limit 202, into an analog or digital signal and then transmits it to the processing unit 31.

[0088] Figure 4An example of a database schema of the model database 203 is shown. In this example, the model database 203 may include a model data table 401 , an input resolution data table 402 , a hardware data table 403 , and a model performance data table 404 . The model data table 401 may include, but is not limited to, fields such as model ID, model name (e.g., VGG19, VGG16, Resnet101, MobileNetV1 / V2), and model size (e.g., 574MB, 528MB, 170MB, 16MB). The input resolution data table 402 may include, but is not limited to, fields such as resolution ID, width (e.g., 640, 512, 416, 320, 256), and height (640, 512, 416, 320, 256). The hardware data table 403 may include, but is not limited to, hardware ID, hardware name (e.g., Intel Core i5-12400, GeForce RTX 4090, AMD Radeon RX 7000, KL520, EdgeTPU, Intel VPU, Movidius Neural Compute Stick), memory size (e.g., 12GB, 8GB, 2GB, 512MB), and clock frequency (e.g., 3.46GHz, 2.5GHz, 1.5GHz, 1.33GHz), the model performance data table 404 may include fields such as performance ID, model ID, hardware ID, resolution ID, latency (e.g., 20ms, 50ms, 80ms, 120ms), and accuracy (e.g., 0.58, 0.62, 0.74, 0.83).

[0089] Model performance data table 404 can be associated with model data table 401, hardware data table 403, and input resolution data table 402 via the model ID field, hardware ID field, and resolution ID field, respectively. Thus, hyperparameter multiplexer 210 can search model database 203 and select a model-related hyperparameter combination from hyperparameter set 204 that meets the hardware specification and performance constraints. For example, if the hardware specification constraint is 512MB of memory for the edge device "KL520," and the corresponding model size of VGG19 is 574MB, exceeding 512MB, VGG19 is excluded from the search. For another example, if the performance constraint is 100 milliseconds, and the Resnet18 model uses an input resolution larger than 416*416, the latency exceeds 100 milliseconds, so input resolutions larger than 416*416 are excluded from the search.

[0090] Figure 5 is a tree diagram showing an exemplary embodiment of the hyperparameter set 204. Figure 5As shown, the hyperparameter set 204 may include multiple sets of model-related hyperparameters 51, such as a set of model architectures 511 (e.g., VGG19, VGG16, Resnet101, MobileNetV1 / V2, etc.) and a set of input resolutions (e.g., 640x640, 512x512, 416x416, 320x320, 256x256, etc.). The hyperparameter set 204 may also include multiple sets of model-independent hyperparameters 52, which can be divided into multiple subsets, such as a training-related subset 520, a loss function-related subset 530, a visual attribute-related subset 540, and a data augmentation-related subset 550. Each of these subsets includes one or more sets of model-independent hyperparameters 52. For example, the training-related subset 520 may include, but is not limited to, a set of learning rates 521 (e.g., 0.005, 0.01, 0.015, 0.02, etc.), a set of momentum 522 (e.g., 0.99, 0.9, 0.8, 0.7, etc.), a set of training cycles (e.g., 10, 20, 30, 50, 100, etc.), and the loss function-related subset may include, but is not limited to, a set of loss function forms (e.g., mean square error (MSE), mean absolute error (MAE), cross-entropy (CE), Kullback-Leibler divergence (KL-D)). divergence)...etc.) and a set of weight combinations (e.g., (α, β) = (0.1, 0.9), (0.2, 0.8), (0.3, 0.7)...etc.), the visual attribute related subset 540 may include but is not limited to a set of color spaces (e.g., HSV, RGB, L*a*b*...etc.) and a set of average brightness 542 of the input image (e.g., 80, 120, 150...etc.), the data enhancement related subset 550 may include but is not limited to a set of rotation angles 551 (e.g., 5°, 15°, 30°, 50°...etc.), a set of translation amounts (e.g., 5 pixels, 10 pixels, 20 pixels...etc.), a set of flip modes (e.g., horizontal flip and vertical flip) and a set of scaling ratios 554 (e.g., 75%, 90%, 125%, 150%...etc.).

[0091] It should be understood that Figure 5The illustrated set of hyperparameters 204 is provided as an example only and is not intended to be limiting. In various implementations, the training subset 520, the loss function subset 530, the visual attribute subset 540, and the data augmentation subset 550 may each include more sets of hyperparameters. For example, the training subset 520 may further include a set of warmup epochs and a set of warmup learning rates, the loss function subset 530 may further include a set of regularization types (e.g., L1 and L2 regularization), and the data augmentation subset 550 may further include a set of shear factors, a set of augmented data percentages (e.g., 1%, 3%, 5%, 2% of the total data for rotated data, 2%, 3%, 5%, 2% of the total data for translated data, 2%, 2%, 2%, and other specific percentages for scaled data), and a set of noise types (e.g., Gaussian noise and salt and pepper noise).

[0092] Figure 6 FIG. 6 is a flow chart of a hyperparameter optimization method 600 according to an embodiment of the present disclosure. Figure 6 As shown, the hyperparameter optimization method 600 may include steps S601 - S603 .

[0093] In step S601, a model-related hyperparameter combination that meets the hardware specification and performance constraints is selected from the hyperparameter set 204 by searching the model database 203. For example, a combination of a model architecture 511 (e.g., Resnet101) and an input resolution 512 (e.g., 416x416) that meets the hardware specification and performance constraints is selected from the hyperparameter set 204.

[0094] In step S602, based on the model-dependent hyperparameter combination, the hyperparameter adjuster 110 searches for the best hyperparameters in each set of model-independent hyperparameters to obtain the best hyperparameter combination. Continuing with the previous example, where the model-dependent hyperparameter combination is composed of the model architecture "Resnet101" and the input resolution "416X416", the hyperparameter multiplexer 210 causes the hyperparameter adjuster 110 to search for the best hyperparameters in each set of model-independent hyperparameters under the conditions of the model architecture being "Resnet101" and the input resolution being "416X416", for example Figure 5As shown in , the best learning rate in the group of learning rates 521, the best momentum in the group of momentum 522, the best training cycle in the group of training cycles 523, the best loss function form in the group of loss function forms, the best weight combination in the group of weight combinations 532, the best color space in the group of color spaces 541, the best average brightness in the group of average brightness 542, the best rotation angle in the group of rotation angles 551, the best translation in the group of translation 552, the best flip mode in the group of flip mode 553, and the best scaling ratio in the group of scaling ratios are obtained, thereby obtaining the best hyperparameter combination consisting of the model architecture "Resnet101", the input resolution "416X416" and the various optimal hyperparameters listed above (such as the best learning rate, the best momentum, the best training cycle, etc.).

[0095] In step S603, a machine learning model is established based on the optimal hyperparameter combination. Like the aforementioned model training program 120, step S603 may also involve the gradient calculation of the loss function and the iterative update of the model parameters, such as the combination of various gradient descent methods and backpropagation algorithms, which are not described in detail here. It should be noted that the machine learning model established in step S603 will be deployed in actual application scenarios, unlike the prototype model 103 trained by the model training program 120 which is only used to evaluate hyperparameters. In addition, the training data set used in step S603 can be the training data set 101, or other larger data sets can be used, and the present disclosure is not limited to this.

[0096] Figure 7 FIG. 7 is a flow chart of a hyperparameter optimization method 700 according to another embodiment of the present disclosure. Figure 6 The hyperparameter optimization method 600 shown in FIG. 700 further includes additional steps S701 - S703 between steps S602 and S603 .

[0097] In step S701, the model performance index corresponding to the optimal hyperparameter combination is tested to see whether it reaches a specified threshold, and a test result is obtained. The model performance index can be, for example, accuracy, precision, recall, F1 score, area under the receiver operating characteristic curve (ROC-AUC), or measured by other custom objective functions. The specified threshold can be set to an appropriate value based on actual needs, and this disclosure is not limited to this.

[0098] In step S702, based on the test results, another model-related hyperparameter combination is selected from the multiple sets of model-related hyperparameters 51. The selection of another model-related hyperparameter combination can depend on practical requirements, in addition to meeting basic model performance, and what further adjustments are desired. For example, if it is desired to further reduce computing costs or increase computing speed while meeting basic accuracy requirements, then if the accuracy does not reach a specified threshold, an attempt can be made to adjust model-related hyperparameters that can improve accuracy. If the model performance indicator has reached the specified threshold, an attempt can be made to adjust model-related hyperparameters that may affect computing costs or computing speed.

[0099] In step S703 , based on the selected another model-related hyperparameter combination, the hyperparameter adjuster 110 searches for the best hyperparameters in each set of model-independent hyperparameters to update the best hyperparameter combination.

[0100] The above steps S701-S703 can be repeatedly executed multiple times until the convergence condition is achieved, that is, the hyperparameter multiplexer 210 determines that it is no longer possible to continue adjusting the model-related hyperparameters to simultaneously meet the model performance and pursue other aspects, such as further reducing computing costs or increasing computing speed.

[0101] In one embodiment, the model-related hyperparameter combination selected in step S601 is a combination of the smallest model architecture 511 and the largest input resolution that meets hardware specification and performance constraints. Subsequently, in step 702, if the test result indicates that the model performance index meets the specified threshold, a lower input resolution 512 is selected from the multiple sets of model-related hyperparameters 51 to form another model-related hyperparameter combination with the previously selected model architecture 511. If the test result indicates that the model performance index does not meet the specified threshold, a larger model architecture 511 is selected from the multiple sets of model-related hyperparameters 51 to form another model-related hyperparameter combination with the previously selected input resolution 512. For example, if a set of model architectures 511 and a set of input resolutions 512 that meet the hardware specification limitations and performance limitations are "VGG19, VGG16, Resnet101, MobileNetV2" and "640X640, 512X512, 416X416, 320X320, 256X256" respectively, in step S601, the smallest "MobileNetV2" in the set of model architectures 511 and the largest "640X640" in the set of input resolutions 512 are selected to form a model-related hyperparameter combination. Then, in step 702, if the test result shows that the model performance index reaches the specified threshold, it means that there is room for reducing the input resolution. Therefore, the hyperparameter multiplexer 21 will select the input resolution "320X320" which is lower than "640X640" and form another model-related hyperparameter combination with the previously selected model architecture "MobileNetV2". If the test result shows that the model performance index does not reach the specified threshold, it means that the model performance should be improved by expanding the model architecture 511. Therefore, the hyperparameter multiplexer 21 will select the model architecture "Resnet101" which is larger than "MobileNetV2" and form another model-related hyperparameter combination with the previously selected input resolution "640X640". The above process can be repeated in multiple iterations of steps S701-S703, and finally the minimum model architecture and input resolution that can meet the performance requirements can be selected.

[0102] In one embodiment, in step S602, the optimal hyperparameters of each set of model-independent hyperparameters in each model-independent hyperparameter subset are searched simultaneously in the order of the training-related subset 520, the loss function-related subset 530, the visual attribute-related subset 540, and the data enhancement-related subset 550. For example, assuming that the model-related hyperparameter combination is composed of the model architecture "Resnet101" and the input resolution "416X416", the hyperparameter multiplexer 210 will cause the hyperparameter adjuster 110 to first search for the optimal learning rate, optimal momentum, and optimal training period in the training-related subset 520 under the conditions of the model architecture being "Resnet101" and the input resolution being "416X416", then search for the optimal loss function shape and optimal weight combination in the loss function-related subset 530, then search for the optimal color space and the optimal average brightness of the input image in the visual attribute-related subset 540, and finally search for the optimal rotation angle, optimal translation, optimal flip mode, and optimal scaling ratio in the data augmentation-related subset. Since the groups of model-independent hyperparameters belonging to the same subset are highly correlated, treating them as a combination and searching them together can further improve the efficiency of hyperparameter optimization.

[0103] The hyperparameter optimization solution provided by the present disclosure can reduce the required search dimensions by grouping hyperparameters to avoid the curse of dimensionality and improve search efficiency. Moreover, by searching for model-related hyperparameters and model-independent hyperparameters together, it can be ensured that they can converge to the global optimal solution together. In addition, by evaluating the model size and input resolution upper limit supported by the hardware, it can be ensured that the searched hyperparameters can meet the model performance requirements while also allowing the hardware to achieve a suitable inference speed. It can also help find the minimum model architecture and input resolution that can meet the performance requirements under limited hardware conditions, taking into account multiple requirements such as model performance, inference speed, and hardware resources.

[0104] The above paragraphs describe various embodiments. Clearly, the teachings herein can be implemented in a variety of ways, and any specific architecture or functionality disclosed in the examples is merely representative. Based on the teachings herein, it should be understood by those skilled in the art that each of the embodiments disclosed herein can be implemented independently, or in combination to implement two or more embodiments.

[0105] Although the present invention has been described above with reference to the embodiments, they are not intended to limit the present invention. Any person skilled in the art may make slight changes and modifications without departing from the concept and scope of the present invention. Therefore, the scope of protection of the invention shall be determined by the claims.

Claims

1. A hyperparameter optimization system comprising: a storage unit storing a program, a model database, and a hyperparameter set, wherein the hyperparameter set includes multiple sets of model-related hyperparameters and multiple sets of model-independent hyperparameters; and A processing unit loads the program from the storage device to execute the following steps: selecting, from the hyperparameter set, a model-related hyperparameter combination that meets a hardware specification constraint and a performance constraint by searching the model database, wherein the model-related hyperparameter combination comprises a model-related hyperparameter from each of the plurality of sets of model-related hyperparameters; Based on the model-dependent hyperparameter combination, searching for an optimal hyperparameter among each set of model-independent hyperparameters using a hyperparameter adjuster to obtain an optimal hyperparameter combination, wherein the optimal hyperparameter combination is composed of the optimal hyperparameters of each of the multiple sets of model-independent hyperparameters and the model-dependent hyperparameter combination; and Based on the optimal hyperparameter combination, a machine learning model is established.

2. The system of claim 1 , wherein the processing unit further performs the following steps: Testing whether a model performance indicator corresponding to the optimal hyperparameter combination reaches a specified threshold, and obtaining a test result; Based on the test result, selecting another model-related hyperparameter combination from the multiple sets of model-related hyperparameters; and Based on the selected another model-related hyperparameter combination, the hyperparameter adjuster searches for the optimal hyperparameter in each group of model-independent hyperparameters to update the optimal hyperparameter combination.

3. The system of claim 2, wherein the model-related hyperparameter combination is a combination of a minimum value among a set of model architectures and a maximum value among a set of input resolutions that meet the hardware specification constraint and the performance constraint; wherein if the model performance indicator reaches the specified threshold, the another model-related hyperparameter combination selected from the multiple sets of model-related hyperparameters has a lower input resolution than the model-related hyperparameter combination; and If the model performance indicator does not reach the specified threshold, the other model-related hyperparameter combination selected from the multiple sets of model-related hyperparameters has a larger model architecture than the model-related hyperparameter combination.

4. The system of claim 1 , wherein the hyperparameter set comprises a plurality of model-independent hyperparameter subsets, the model-independent hyperparameter subsets comprising a training-related subset, a loss function-related subset, a visual attribute-related subset, and a data augmentation-related subset, and each of the model-independent hyperparameter subsets comprises one or more of the plurality of sets of model-independent hyperparameters; and The processing unit further searches for the optimal hyperparameters of each group of model-independent hyperparameters in each model-independent hyperparameter subset through the hyperparameter adjuster in the order of the training-related subset, the loss function-related subset, the visual attribute-related subset and the data enhancement-related subset.

5. The system of claim 4, wherein the training-related subset includes a set of learning rates, a set of momentums, and a set of training cycles.

6. The system of claim 4, wherein the loss function related subset includes a set of loss function shapes and a set of weight combinations. The system of claim 4 , wherein the visual attribute related subset comprises a set of color spaces and a set of average brightness.

8. The system of claim 4, wherein the data enhancement related subset includes a set of rotation angles, a set of translation amounts, a set of flip modes, and a set of scaling ratios. 9 . The system of claim 1 , wherein the hardware specification restriction comprises at least one of a memory capacity, a clock frequency, and a maximum number of supported channels of a target hardware.

10. The system of claim 1, wherein the performance constraint comprises at least one of a specified delay value and a power consumption.

11. A hyperparameter optimization method implemented by a computer system, the method comprising the following steps: selecting, by searching a model database, a model-related hyperparameter combination from a hyperparameter set that satisfies a hardware specification constraint and a performance constraint, wherein the hyperparameter set includes a plurality of sets of model-related hyperparameters and a plurality of sets of model-independent hyperparameters, and wherein the model-related hyperparameter combination is composed of a model-related hyperparameter from each of the plurality of sets of model-related hyperparameters; Based on the model-dependent hyperparameter combination, searching, by a hyperparameter adjuster, for an optimal hyperparameter among each set of model-independent hyperparameters to obtain an optimal hyperparameter combination, wherein the optimal hyperparameter combination is composed of the optimal hyperparameters of each of the multiple sets of model-independent hyperparameters and the model-dependent hyperparameter combination; and Based on the optimal hyperparameter combination, a machine learning model is established.

12. The method according to claim 11, further comprising the steps of: Testing whether a model performance indicator corresponding to the optimal hyperparameter combination reaches a specified threshold, and obtaining a test result; Based on the test result, selecting another model-related hyperparameter combination from the multiple sets of model-related hyperparameters; as well as Based on the selected another model-related hyperparameter combination, the hyperparameter adjuster searches for the optimal hyperparameter in each group of model-independent hyperparameters to update the optimal hyperparameter combination.

13. The method of claim 12, wherein the model-related hyperparameter combination is a combination of a minimum among a set of model architectures and a maximum among a set of input resolutions that meet the hardware specification constraint and the performance constraint; wherein if the model performance indicator reaches the specified threshold, the another model-related hyperparameter combination selected from the multiple sets of model-related hyperparameters has a lower input resolution than the model-related hyperparameter combination; and If the model performance indicator does not reach the specified threshold, the other model-related hyperparameter combination selected from the multiple sets of model-related hyperparameters has a larger model architecture than the model-related hyperparameter combination.

14. The method of claim 11 , wherein the hyperparameter set comprises a plurality of model-independent hyperparameter subsets, the model-independent hyperparameter subsets comprising a training-related subset, a loss function-related subset, a visual attribute-related subset, and a data augmentation-related subset, and the model-independent hyperparameter subsets each comprise one or more sets of the plurality of model-independent hyperparameter sets; The step of searching for the optimal hyperparameters in each group of model-independent hyperparameters based on the model-related hyperparameter combination by the hyperparameter adjuster to obtain the optimal hyperparameter combination further includes: According to the order of the training-related subset, the loss function-related subset, the visual attribute-related subset and the data enhancement-related subset, the optimal hyperparameters of each group of model-independent hyperparameters in each model-independent hyperparameter subset are searched at the same time by the hyperparameter adjuster. The method of claim 14 , wherein the training-related subset comprises a set of learning rates, a set of momentums, and a set of training cycles. The method of claim 14 , wherein the loss function related subset comprises a set of loss function shapes and a set of weight combinations. The method of claim 14 , wherein the visual attribute related subset comprises a set of color spaces and a set of average brightness.

18. The method of claim 14, wherein the data enhancement related subset comprises a set of rotation angles, a set of translation amounts, a set of flip modes, and a set of scaling ratios. 19 . The method of claim 11 , wherein the hardware specification restriction comprises at least one of a memory capacity, a clock frequency, and a maximum number of supported channels of a target hardware.

20. The method of claim 11, wherein the performance constraint comprises at least one of a specified delay value and a power consumption.

Citation Information

Cited By

  • MaaS platform large model reasoning parameter automatic optimization acceleration method

    CN120950083A

  • MaaS platform large model inference parameter automatic optimization and acceleration method

    CN120950083B