Automatic optimization method of support vector machine hyperparameters
By optimizing the hyperparameters of support vector machines using improved whale and bat optimization algorithms, the problems of high computational cost and susceptibility to local optima are solved. This achieves efficient and stable hyperparameter combination search, improving the model's classification accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LINKER
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-19
AI Technical Summary
Existing hyperparameter optimization methods for Support Vector Machines (SVM) and their extended model, Two Support Vector Machines (TWSVM), suffer from high computational costs, susceptibility to local optima, and unbalanced search performance. Traditional methods such as grid search and random search are inefficient, while metaheuristic algorithms such as particle swarm optimization, genetic algorithms, and bat algorithms suffer from premature convergence and imbalance between global and local searches.
We employ an improved Whale Optimization Algorithm (EWOA) and Bat Algorithm (OSIBA), and optimize the hyperparameter combination of SVM and TWSVM by introducing Sobol sequence initialization, piecewise nonlinear convergence factor, adaptive weight strategy and hybrid perturbation mechanism, and combining K-fold cross-validation classification accuracy as fitness function.
It significantly reduces computational costs, improves hyperparameter optimization efficiency, avoids premature convergence and local optima, and enhances the model's generalization ability and classification performance.
Smart Images
Figure CN122241395A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning, and more specifically to an automatic hyperparameter optimization method for support vector machines. Background Technology
[0002] Support Vector Machine (SVM) and its extended model, Two Support Vector Machine (TWSVM), are high-performance classification algorithms in the field of machine learning. SVM exhibits good generalization ability in classification and regression tasks, while TWSVM significantly improves training speed by constructing hyperplanes for the two classes, decomposing the large quadratic programming problem into two smaller problems. However, the performance of both is highly dependent on the reasonable selection of hyperparameters. SVM mainly relies on the penalty coefficient C and the kernel function parameter g, while TWSVM needs to optimize the first-class penalty parameter C1, the second-class penalty parameter C2, and the kernel function parameter g. Improper parameter selection will directly lead to a decrease in the model's generalization ability and an increase in classification error. Traditional hyperparameter optimization methods, such as grid search and random search, suffer from high computational costs, low efficiency, and susceptibility to local optima. While emerging metaheuristic algorithms (such as Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Bat Algorithm (BA), and Whale Algorithm (WOA)) have improved the automation of parameter optimization to some extent, they still have their limitations: standard algorithms like PSO and GA are prone to premature convergence and insufficient search accuracy in later stages; BA, while characterized by fewer parameters, simpler structure, and strong local optimization capabilities, requires further balancing of global and local search performance; and WOA suffers from strong dependence on initial solutions, loss of population diversity in later iterations, and imbalance between global and local searches. Summary of the Invention
[0003] To address the shortcomings of existing technologies, the present invention aims to provide an automatic hyperparameter optimization method for support vector machines. By improving the whale optimization algorithm and the bat algorithm, the method solves the problems of high computational cost, easy getting trapped in local optima, and unbalanced search performance of traditional hyperparameter optimization methods.
[0004] To achieve the above objectives, the present invention provides the following technical solution: an automatic hyperparameter optimization method for support vector machines, comprising the following steps: Step 1: Construct and initialize the support vector machine model or double support vector machine model, and initialize the population for the optimization algorithm; Step 2: Use the K-fold cross-validation classification accuracy of the support vector machine model or the dual support vector machine model as the fitness function, and then use the optimization algorithm in step 1 to iteratively search for hyperparameter combinations. Step 3: Output the hyperparameter combination that optimizes the fitness function obtained in Step 2; Step four: Use the optimal hyperparameter combination obtained in step three to train the final support vector machine model and use it for classification prediction, or use the dual support vector machine classification model and use it for prediction of the samples to be classified. The optimization algorithm is either the whale optimization algorithm or the bat optimization algorithm. When the optimization algorithm is the whale optimization algorithm, the hyperparameters include the penalty coefficient C and the kernel function parameter. The hyperparameter combination is (C, When the optimization algorithm is the bat algorithm, the hyperparameters include the first type of penalty parameter. Second type of penalty parameter and kernel function parameters The hyperparameter combination is .
[0005] As a further improvement of the present invention: when the optimization algorithm in step one is the whale optimization algorithm, the population is initialized using the Sobol sequence.
[0006] As a further improvement of the present invention: the whale optimization algorithm adopts a piecewise nonlinear convergence factor. Its expression is: As a further improvement of the present invention: the whale optimization algorithm introduces an adaptive weight strategy during the position update process, and its position update formula is as follows: As a further improvement of the present invention, the whale optimization algorithm introduces a hybrid perturbation mechanism, including a population perturbation based on the firefly algorithm and an individual perturbation based on Gaussian mutation, and uses a greedy principle to retain a better solution.
[0007] As a further improvement of the present invention: when the optimization algorithm in step one is the bat algorithm, its speed update formula is: Wherein, the inertial weight The strategy is dynamically adjusted based on the optimal success rate.
[0008] As a further improvement of the present invention: the inertial weight The dynamic adjustment strategy is as follows: in, For the first The optimal success rate of the generation population. Indicators for judging the merits or demerits of individuals. For the first The individual in the first The fitness value of the generation.
[0009] As a further improvement of the present invention: the bat algorithm, in its local search phase, employs a hybrid perturbation strategy based on Gaussian and Cauchy distributions to generate new solutions, the formula of which is: in, This represents the current iteration number. The maximum number of iterations, These are standard Cauchy distribution random numbers. These are random numbers distributed according to a standard Gaussian distribution.
[0010] The beneficial effects of this invention lie in its application of the whale optimization algorithm and the bat algorithm to support vector machine hyperparameter optimization, using K-fold cross-validation classification accuracy as the fitness function, thus achieving efficient search for hyperparameter combinations. Compared to traditional grid search and random search, it significantly reduces computational costs and improves optimization efficiency; compared to standard metaheuristic algorithms, the improved optimization strategy effectively avoids premature convergence and local optima problems, enhancing the model's generalization ability. Attached Figure Description
[0011] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 A comparison of the convergence curves of EWOA and WOA on the Sphere function; Figure 3 A comparison of the classification accuracy of EWOA-SVM and WOA-SVM on multiple datasets; Figure 4 This is the overall flowchart of the OSIBA-TWSVM model construction and optimization of the present invention.
[0012] Figure 5 This is a bar chart comparing the classification accuracy of OSIBA-TWSVM and traditional BA-TWSVM on representative datasets in this embodiment of the invention.
[0013] Figure 6 This is a graph comparing the average classification accuracy of OSIBA-TWSVM with other heuristically optimized TWSVM on multiple datasets in this embodiment of the invention. Detailed Implementation
[0014] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.
[0015] The automatic hyperparameter optimization method for support vector machines in this embodiment includes the following steps: Step 1: Construct and initialize the support vector machine model or double support vector machine model, and initialize the population for the optimization algorithm; Step 2: Use the K-fold cross-validation classification accuracy of the support vector machine model or the dual support vector machine model as the fitness function, and then use the optimization algorithm in step 1 to iteratively search for hyperparameter combinations. Step 3: Output the hyperparameter combination that optimizes the fitness function obtained in Step 2; Step four: Use the optimal hyperparameter combination obtained in step three to train the final support vector machine model and use it for classification prediction, or use the dual support vector machine classification model and use it for prediction of the samples to be classified. The optimization algorithm is either the whale optimization algorithm or the bat optimization algorithm. When the optimization algorithm is the whale optimization algorithm, the hyperparameters include the penalty coefficient C and the kernel function parameter. The hyperparameter combination is (C, When the optimization algorithm is the bat algorithm, the hyperparameters include the first type of penalty parameter. Second type of penalty parameter and kernel function parameters The hyperparameter combination is This method addresses the high computational cost of traditional methods by combining optimization algorithms with cross-validation: Step 1 initializes the population to ensure diversity of the search starting point; Step 2 guides the iteration direction with classification accuracy as the objective function; Step 3 outputs the globally optimal parameter combination; and Step 4 completes model training and prediction. Through the targeted design of two optimization algorithms, adapted to the parameter characteristics of SVM and TWSVM respectively, it effectively improves the efficiency and accuracy of hyperparameter optimization, avoiding the subjectivity of manual parameter tuning and the trap of local optima.
[0016] Reference Figures 1 to 3 As shown, this embodiment provides an Example 1, which further details the whale optimization algorithm applied based on the above method. The existing whale optimization algorithm is a swarm intelligence optimization algorithm that simulates the feeding behavior of humpback whales. It has advantages such as simple structure and fast convergence speed, but still suffers from drawbacks such as strong dependence on initial solutions, loss of population diversity in the later stages of iteration, and imbalance between global and local searches. In recent years, scholars have proposed various improvement strategies, such as chaotic mapping initialization, nonlinear convergence factors, adaptive weights, and hybrid perturbation mechanisms, to improve its optimization performance.
[0017] Based on existing technology, the Enhanced Whale Optimization Algorithm (EWOA) of this embodiment is constructed by introducing improvements such as Sobol sequence initialization, piecewise nonlinear convergence factor, adaptive weight strategy, and hybrid perturbation mechanism on the basis of traditional WOA. This improved algorithm is creatively applied to the SVM hyperparameter optimization problem to achieve efficient and stable automatic parameter optimization. Among these improvements, the piecewise nonlinear convergence factor... Its expression is: An adaptive weighting strategy is introduced during the location update process, and the location update formula is as follows: A hybrid perturbation mechanism is introduced, including population perturbation based on the firefly algorithm and individual perturbation based on Gaussian mutation, and a greedy principle is used to preserve the better solution; thus, the specific implementation process is as follows: Step 1: Load the dataset and normalize it, then divide the training set and test set in a 9:1 ratio; Step 2: Set the SVM hyperparameter search range: C [0.01,1000], g [0.01,10]; Step 3: Initialize EWOA algorithm parameters: population size N=10, maximum number of iterations. =50; Step 4: Run the EWOA algorithm, using the accuracy of 3-fold cross-validation as the fitness function, and iteratively optimize. Step 5: Output the optimal hyperparameters, train the final SVM model, and test it.
[0018] Experimental results show that the method of the present invention achieves a classification accuracy of 78.0% on the "German" dataset, which is significantly better than the 72.9% of the traditional WOA-SVM, thus verifying the effectiveness of the present invention.
[0019] Reference Figures 4 to 6 As shown, this embodiment provides an Example 1, which further details the applied Bat Algorithm based on the above method. The Bat Algorithm (BA) is a swarm intelligence optimization algorithm that simulates the echolocation behavior of bats, characterized by few parameters, simple structure, and strong local exploitation capabilities. Ding Yuanming et al. proposed an improved Bat Algorithm (OS-PSOBA, abbreviated as OSIBA in this invention) in their paper "Improved Bat Algorithm for UAV Path Planning." This algorithm expands the search range by introducing individual optimal factors from the particle swarm optimization algorithm, uses a Gaussian-Cauchy fusion distribution for local fine-grained search, and dynamically adjusts the inertia weight using an optimal success rate strategy, effectively balancing global exploration and local exploitation capabilities. It demonstrates excellent optimization performance in UAV path planning problems. The speed update formula in the aforementioned Bat Algorithm is: Wherein, the inertial weight The strategy is dynamically adjusted based on the optimal success rate.
[0020] Inertia weight The dynamic adjustment strategy is as follows: In the local search phase of the Bat algorithm, a hybrid perturbation strategy based on Gaussian and Cauchy distributions is used to generate new solutions. Its update formula is: Generate a random number , This bat algorithm is the first to introduce the efficient OSIBA optimization algorithm into the field of TWSVM parameter optimization, providing a novel and non-obvious technical approach to solving this problem. Because the OSIBA algorithm itself possesses powerful global exploration and local exploitation capabilities, its application to TWSVM parameter optimization effectively avoids the tendency of traditional methods to get trapped in local optima, quickly and accurately finding the globally optimal or near-optimal solution in a complex parameter space. Test results of the OSIBA-optimized TWSVM model on multiple standard datasets show that its classification accuracy, precision, recall, and F1 score are significantly better than TWSVM models optimized by traditional optimization algorithms (such as standard PSO, BA) or other improved algorithms. Furthermore, the described method has a clear flow and can be automated, greatly reducing reliance on manual parameter tuning experience, providing reliable technical support for the high-performance application of TWSVM in image recognition, medical diagnosis, financial risk control, and other fields. The specific implementation process is as follows: Step 1: Data preprocessing. Load the "Breast Cancer" dataset and normalize the feature data to eliminate the influence of units. Randomly shuffle the dataset and divide it into training and test sets in a 9:1 ratio.
[0021] Step 2: Set the search range for TWSVM hyperparameters: , Initialize OSIBA algorithm parameters: population size Maximum number of iterations Inertial weight range acceleration coefficient Frequency range Loudness attenuation coefficient Pulse enhancement coefficient The fitness function is defined as the average classification accuracy on the training set obtained through 3-fold cross-validation.
[0022] Step 3: Combining the hyperparameters of TWSVM The location of the bats is mapped to the position vector in the OSIBA algorithm. The OSIBA algorithm is run, iteratively updating the bat population's location based on the aforementioned velocity update formula, local search strategy, and dynamic adjustment strategy using inertia weights. In each generation, a TWSVM model is trained based on the bats' locations (i.e., a set of hyperparameters), and its 3-fold cross-validation accuracy is calculated as the fitness value. The optimal location of each individual is recorded. and global optimal position After iterating to the maximum algebraic number, the globally optimal position, i.e., the optimal combination of hyperparameters, is output. .
[0023] Step 4: Use the obtained optimal hyperparameters Retrain a final TWSVM model on the entire training set. Evaluate the model's performance using the test set, recording metrics such as classification accuracy, precision, recall, and F1 score. Step Six: The OSIBA-TWSVM method proposed in this invention achieved a classification accuracy of 93.16% on the "Breast Cancer" dataset, which is superior to the 93.10% of the traditional BA-TWSVM and other comparative algorithms. This result confirms the effectiveness and superiority of this invention in improving the classification performance of TWSVM.
[0024] In summary, this invention provides an automatic hyperparameter optimization method for Support Vector Machines (SVMs). Through improved whale and bat optimization algorithms, and using K-fold cross-validation with classification accuracy as the fitness function, it achieves efficient search for hyperparameter combinations. This scheme addresses the problems of high computational cost, susceptibility to local optima, and unbalanced search performance in traditional optimization methods. Through improvements such as Sobol sequence initialization, piecewise convergence factors, adaptive weights, and hybrid perturbation mechanisms, it significantly improves the efficiency, accuracy, and stability of hyperparameter optimization, enabling SVM models to exhibit superior generalization ability and classification performance in classification tasks.
[0025] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. An automatic hyperparameter optimization method for support vector machines, characterized in that: Includes the following steps: Step 1: Construct and initialize the support vector machine model or double support vector machine model, and initialize the population for the optimization algorithm; Step 2: Use the K-fold cross-validation classification accuracy of the support vector machine model or the dual support vector machine model as the fitness function, and then use the optimization algorithm in step 1 to iteratively search for hyperparameter combinations. Step 3: Output the hyperparameter combination that optimizes the fitness function obtained in Step 2; Step four: Use the optimal hyperparameter combination obtained in step three to train the final support vector machine model and use it for classification prediction, or use the dual support vector machine classification model and use it for prediction of the samples to be classified. The optimization algorithm is either the whale optimization algorithm or the bat optimization algorithm. When the optimization algorithm is the whale optimization algorithm, the hyperparameters include the penalty coefficient C and the kernel function parameter. The hyperparameter combination is (C, When the optimization algorithm is the bat algorithm, the hyperparameters include the first type of penalty parameter. Second type of penalty parameter and kernel function parameters The hyperparameter combination is .
2. The automatic hyperparameter optimization method for support vector machines according to claim 1, characterized in that: When the optimization algorithm in step one is the whale optimization algorithm, the population is initialized using the Sobol sequence.
3. The automatic hyperparameter optimization method for support vector machines according to claim 2, characterized in that: The whale optimization algorithm employs a piecewise nonlinear convergence factor. Its expression is: 。 4. The automatic hyperparameter optimization method for support vector machines according to claim 3, characterized in that: The whale optimization algorithm introduces an adaptive weight strategy during the position update process, and its position update formula is as follows: 。 5. The automatic hyperparameter optimization method for support vector machines according to claim 4, characterized in that: The whale optimization algorithm introduces a hybrid perturbation mechanism, including a population perturbation based on the firefly algorithm and an individual perturbation based on Gaussian mutation, and uses a greedy principle to retain the better solution.
6. The automatic hyperparameter optimization method for support vector machines according to claim 1, characterized in that: When the optimization algorithm in step one is the Bat Algorithm, its speed update formula is: Wherein, the inertial weight The strategy is dynamically adjusted based on the optimal success rate.
7. The automatic hyperparameter optimization method for support vector machines according to claim 6, characterized in that: The inertial weight The dynamic adjustment strategy is as follows: in, For the first The optimal success rate of the generation population. Indicators for judging the merits or demerits of individuals. For the first The individual in the first The fitness value of the generation.
8. The automatic hyperparameter optimization method for support vector machines according to claims 6 to 7, characterized in that: The bat algorithm, in its local search phase, employs a hybrid perturbation strategy based on Gaussian and Cauchy distributions to generate new solutions, the formula of which is: in, This represents the current iteration number. The maximum number of iterations, These are standard Cauchy distribution random numbers. These are random numbers distributed according to a standard Gaussian distribution.