A Kernel Ensemble Support Vector Machine Image Classification Method Based on a Shared Parameter Space

Abstract

The present invention discloses a kernel ensemble support vector machine image classification method based on a shared parameter space, which obtains image data to form a sample set, preprocesses the sample set, and uses the kernel method to obtain different individual basic kernels of the sample set; constructs a single-kernel model based on the individual basic kernels; uses constraints to integrate the losses of multiple single-kernel models into a unified loss, and this unified loss is the ensemble loss, and then constructs a kernel ensemble support vector machine model in the shared parameter space by multiple single-kernel models; further learns the common parameters and individual specific parameters shared by each kernel in multiple kernel spaces; uses the kernel ensemble support vector machine model in the shared parameter space to train a classifier to classify the image to be classified and obtain a classification result. The image classification method proposed by the present invention can improve the accuracy of image classification.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and image processing, and particularly relates to an image classification method based on a kernel integrated support vector machine with a shared parameter space. Background Art

[0002] A support vector machine (SVM) is a type of generalized linear classifier that performs binary classification on data in a supervised learning manner. Its basic idea is to construct an optimal hyperplane in the sample input space or feature space, maximizing the distance between the hyperplane and two types of sample sets, thereby obtaining the best generalization ability. SVM uses a hinge loss function to calculate the empirical risk and adds a regularization term to the solution model to optimize the structural risk. It is a classifier with sparsity and robustness. For non-linear classification problems, SVM attempts to transform them into linear problems in another space through non-linear transformation (kernel function) and solve the optimal linear classification surface in this transformed space. This non-linear transformation can be achieved by defining an appropriate inner product function, that is, the kernel function.

[0003] The kernel function can map features from a low-dimensional space to a high-dimensional space. However, currently, the commonly used SVMs are all single-kernel, that is, they are based on a single feature space. In practical applications, it is often necessary to select different kernel functions (such as Gaussian kernel function, polynomial kernel function, etc.) according to experience and specify different parameters. This is not only inconvenient but also the features in practical applications are often not single but heterogeneous. The best kernel functions corresponding to these heterogeneous features may not be the same. If they share the same kernel function, it may not be possible to obtain the optimal mapping, that is, it may not be possible to obtain a relatively accurate classification result.

[0004] Ensemble learning refers to a machine learning method that uses multiple single-kernel classification models for learning. Then, through the integration and fusion of these single-kernel classification models, a better classification effect than the original single-kernel classification model can be obtained. For image classification, the accuracy of image classification can be improved by integrating multiple single-kernel classification models. However, existing kernel ensemble learning methods basically adopt a unified similarity measure in the entire input space. When a category shows a high degree of difference and correlation with other categories, it is difficult for them to cope with the complexity of data distribution. Summary of the Invention

[0005] In order to solve the deficiencies existing in the prior art, the present application proposes an image classification method based on a kernel integrated support vector machine with a shared parameter space. By designing a unified integration loss, multiple single-kernel classification models are integrated, and by learning the common parameters and individual specific parameters shared by each kernel in multiple kernel spaces, the accuracy of image classification is improved.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An image classification method based on a kernel ensemble support vector machine with a shared parameter space, comprising the following steps:

[0008] Step 1: Obtain image data to form a sample set, preprocess the sample set, and use the kernel method to obtain different individual basic kernels of the sample set;

[0009] Step 2: Construct a single-kernel model based on the individual basic kernels; use constraints to integrate the losses of multiple single-kernel models into a unified loss, and this unified loss is the ensemble loss. Furthermore, construct a kernel ensemble support vector machine model in the shared parameter space from multiple single-kernel models;

[0010] Step 3: Input the training set into the kernel ensemble support vector machine model, and further learn the common parameter α0 and the individual specific parameter α shared by each kernel in multiple kernel spaces i ; the designed kernel ensemble support vector machine model jointly optimizes each single-kernel model by minimizing the ensemble loss; thus, a trained kernel ensemble support vector machine model based on the shared parameter space is obtained;

[0011] Step 4: Set the determination condition for the iteration stop of the kernel ensemble support vector machine model. When the determination condition is met, jump out of the loop; otherwise, continue to execute the loop; when the loop exits, it means that the optimal solution w has been found, thus obtaining a trained kernel ensemble support vector machine model based on the shared parameter space; obtain

[0012]

[0013] where f(x t ) is the predicted label result;

[0014] Step 5: Input the test set X test = [x1, x2,..., x N ∈ R N*q into the trained kernel ensemble support vector machine model in the shared parameter space to obtain the final binary classifier model: Y = sign(f(X test ))), that is, the target classifier, and use the target classifier to classify the image to be classified to obtain the classification result.

[0015] Furthermore, the objective function of the kernel ensemble support vector machine model is expressed as:

[0016]

[0017] S.T.1 T w = 1, w i ≥ 0,

[0018]

[0019] Among them, C is a parameter used to achieve the balance between the minimum empirical loss and the minimum structural loss; w i is used to control the weight of the loss in each reproducing kernel Hilbert space, and w is a vector composed of the weights w i ; K i is the i-th kernel Gram matrix, and α i and α0 are both shared parameters related to the weights of each training data sample in K i and are both column vectors of N×1; L is the number of single base kernels, and b i is the offset of K i ; ξ it is the hinge loss of the t-th sample in the i-th kernel, t = 1, 2,... N, and N is the number of samples; α 0m , α im are respectively the m-th components in the column vectors α i and α0; x t , x m are respectively the t-th sample and the m-th sample.

[0020] Furthermore, the training process of the kernel ensemble support vector machine model is as follows:

[0021] Step1:

[0022] Fix the value of w and initialize Then, the kernel ensemble support vector machine model is transformed into the dual form using the Lagrange multiplier method as follows:

[0023]

[0024]

[0025]

[0026] Among them, β is the column vector formed by the Lagrange multipliers, and β t is the t-th component in the column vector β, t = 1, 2,... N; Y is the diagonal matrix formed by the sample labels y1, y2,..., y N , and y t is the t-th sample label, t = 1, 2,... N; C is the penalty coefficient,

[0027] Since the obtained dual form of the kernel ensemble support vector machine model is the dual form of the standard SVM, the SVM solver is used to solve for the maximum value of β;

[0028] Step2:

[0029] Fix the obtained maximum value of β, and update w by solving the minimization problem. The solution process is as follows:

[0030] Apply the augmented Lagrangian multiplier method to the dual form of the kernel integrated support vector machine model to obtain the following formula:

[0031]

[0032] where A i and B i are weighted combinations of basic kernels, and the corresponding w j is the weight of the j-th single-core model, and the initial parameters are: w i , λ, η i , μ; η i , μ, and λ are all Lagrange multipliers;

[0033] Step3:

[0034] Take the partial derivative of the above formula with respect to w i and set it to zero to obtain the update formula for w i as shown below:

[0035]

[0036] Step4:

[0037] Solve for λ, η i and μ in the following order, where θ is the learning rate;

[0038]

[0039] η i = η i + θ * w i

[0040]

[0041] Step5:

[0042] Based on the solved β and w i , further complete the learning of the common parameter α0 and the individual specific parameter α i , which are respectively expressed as:

[0043]

[0044] α i = w i Yβ

[0045] where K i is the i-th kernel Gram matrix, Kj is the j-th kernel Gram matrix, where j = 1, 2, ..., L, and L is the number of individual base kernels.

[0046] Furthermore, the determination condition for the iteration stop of the kernel ensemble support vector machine model is: when the absolute value of the difference in losses before and after the update, loss, is less than 1e-6 or the number of iterations is greater than 200 times, that is, it meets the determination condition and the iteration stops.

[0047] Furthermore, let loss new = 0 at the beginning, and calculate Thus, we get:

[0048] loss = |loss old - loss new |;

[0049] where loss is the absolute value of the difference in losses before and after the update, loss old is the loss before the update, and loss new is the loss after the update;

[0050] Recalculate according to the updated parameters:

[0051]

[0052] Thus, we get the updated loss = |loss old - loss new |, save loss old ← loss new , and increase the number of iterations by 1.

[0053] Furthermore, the kernel ensemble support vector machine is combined and improved into a classifier applicable to multi-class problems; the one-versus-rest method is used to extend it to solve multi-class problems; the samples of a certain class are regarded as one class, and the rest are uniformly regarded as another class, obtaining the same number of binary classifiers as the number of sample types, and then classifying the samples into the class with a larger test output value to obtain the classification result.

[0054] Furthermore, in Step 2, the initial parameters are: λ = 0, η i = 0, μ = 1.

[0055] Furthermore, 60% of the sample set is randomly selected from the normalized sample set as the training set, and the remaining 40% of the sample set is used as the test set.

[0056] Furthermore, the preprocessing described in Step 1 is to normalize the sample set.

[0057] Advantages of the present invention:

[0058] In the present invention, by designing an integration loss, multiple individual kernels are integrated into a whole, thereby proposing a kernel integration support vector machine model. The design of this integration loss helps to reduce the overhead of the proposed model and improve its accuracy and robustness in an integrated manner.

[0059] In addition, the present invention also introduces shared parameters in multiple kernel spaces, where the common parameters are used to learn the common structure of multiple kernel representations, and the individual specific parameters are used to learn the kernel-specific structure; thereby, the accuracy and robustness of the model proposed by the present invention can be further enhanced. Description of the Drawings

[0060] Figure 1 is a flowchart of an image classification method of a kernel integration support vector machine based on a shared parameter space according to the present invention. Detailed Embodiments

[0061] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] An image classification method of a kernel integration support vector machine based on a shared parameter space includes the following steps:

[0063] Step 1, obtain image data to form a sample set, perform normalization processing on the sample set, and randomly select 60% of the sample set as the training set, and the remaining 40% of the sample set as the test set. For the space R, there is a sample set X = [x1, x2,..., x N ∈ R N*q and the label set Y = [y1, y2,..., y N ∈ R N*p , where, x N is the Nth image sample, y N is the label corresponding to the Nth image sample, N is the number of samples, q is the dimension of X, and p is the dimension of Y.

[0064] Use the kernel method for the training set and the test set respectively, and then obtain different individual basic kernels corresponding to the training set and the test set.

[0065] In this embodiment, the method for obtaining different individual basic kernels according to data samples is: there is a mapping function φ(x) from the input space R to the feature space , and map X from the space R to the high-dimensional feature space , for all \(x, z\in X\), the function \(K(x, z)\) satisfies \(K(x, z)=\langle\varphi(x),\varphi(z)\rangle\), then \(K(x, z)\) is called a kernel function. Define the matrix element \(K\) ij \(=K(x\) i , \(x\) j ), and obtain different single basic kernels by calculating the kernel function values between different samples.

[0066] Step 2: Use constraints to integrate the losses of multiple single - core models into a unified loss, which is the integrated loss. Then, construct a kernel - integrated support vector machine model in the shared parameter space from multiple single - core models. The objective function of this model is expressed as follows:

[0067]

[0068] where \(C\) is a parameter used to achieve the balance between the minimum empirical loss and the minimum structural loss. \(w\) i is the weight used to control the loss in each Reproducing Kernel Hilbert Space (RKHS). \(w\) is a vector composed of weights \(w\) i . \(K\) i is the \(i\) - th kernel Gram matrix. \(\alpha\) i and \(\alpha_0\) are both shared parameters related to the weights of each training data sample in \(K\) i and are both column vectors of size \(N\times1\). \(L\) is the number of single basic kernels. \(b\) i is the offset of \(K\) i . \(\xi\) it is the hinge loss of the \(t\) - th sample in the \(i\) - th kernel, \(t = 1,2,\cdots,N\), and \(N\) is the number of samples. \(\alpha\) 0m , \(\alpha\) im are the \(m\) - th components in the column vectors \(\alpha\) i and \(\alpha_0\) respectively. \(x\) t , \(x\) m are the \(t\) - th sample and the \(m\) - th sample respectively.

[0069] The first term is the unified hinge loss, which is used as a standard to judge the quality of the kernel - integrated support vector machine model. The second term and the third term are combined as the regularization term.

[0070] Step 3: Input the training set into the kernel - integrated support vector machine model, and further learn the common parameter (\(\alpha_0\)) and the single specific parameter (\(\alpha\) i ) shared by each kernel in multiple kernel spaces. The designed kernel - integrated support vector machine model jointly optimizes each single - core model by minimizing the integrated loss. Thus, a trained kernel - integrated support vector machine model based on the shared parameter space is obtained.

[0071] The training process of the kernel integrated support vector machine model is as follows:

[0072] Step1:

[0073] First, fix the value of w and initialize Then, transform the kernel integrated support vector machine model into the dual form using the Lagrange multiplier method as follows:

[0074]

[0075] where β is the column vector formed by the Lagrange multipliers, β t is the t-th component in the column vector β, t = 1, 2,..., N; Y is the diagonal matrix composed of the sample labels y1, y2,..., y N and y t is the t-th sample label, t = 1, 2,..., N; C is the penalty coefficient. Since the obtained formula (2) is the dual form of the standard SVM, the SVM solver can be used to solve for the maximum value of β.

[0076] Step2:

[0077] Fix β obtained in Step1 and update w by solving the minimization problem. The solution process is as follows:

[0078] Apply the augmented Lagrange multiplier method to formula (2) to obtain the following formula:

[0079]

[0080] where A i and B i are the weighted combination of the basic kernels, and the corresponding is the weight of the j-th single-core model, K j is the j-th kernel Gram matrix, j = 1, 2,..., L, where L is the number of single basic kernels; the initial parameters are: λ = 0, η i = 0, μ = 1; η i , μ, and λ are all Lagrange multipliers.

[0081] Step3:

[0082] Take the partial derivative of formula (3) with respect to w i and set it to zero to obtain the update formula for w i shown in formula (4).

[0083]

[0084] Step4:

[0085] Solve for λ and η in the following order i Set μ and θ as learning rates to 1.1

[0086]

[0087] η i = η i + θ * w i (6)

[0088]

[0089] Step5:

[0090] Based on the solved β and w i , further complete the learning of the common parameter (α0) and the individual specific parameter (α i ), which are respectively expressed as:

[0091]

[0092] α i = w i Yβ

[0093] Step 4: Set the decision condition for the iteration stop of the kernel integrated support vector machine model. When the decision condition is met, break out of the loop; otherwise, continue to execute the loop. When the loop exits, it means that the optimal solution w has been found, and thus the trained kernel integrated support vector machine model based on the shared parameter space is obtained. According to the optimal solution w, obtain where f(x t ) is the predicted label result

[0094] In this application, the decision condition is set as:

[0095] According to formula (3), assign initial values to the parameters , set loss new = 0 at the beginning, calculate to obtain:

[0096] loss = |loss old - loss new |;

[0097] where loss is the absolute value of the difference in losses before and after the update, loss old is the loss before the update, and loss new is the loss after the update

[0098] Recalculate according to the updated parameters to obtain the updated loss = |loss old - loss new|, save the loss old ←loss new , increment the iteration count by 1; loop and iterate to solve for w in step 3 i , λ, η i and the process of parameters μ until the loss is less than 1e-6 or the iteration count is greater than 200 times, i.e., meeting the determination condition.

[0099] Step 5, input the test set X test = [x1, x2,..., x N ∈ R N*q into the kernel integrated support vector machine model in the trained shared parameter space to obtain the final binary classifier model: Y = sign(f(X test ))), i.e., the target classifier. Correspondingly, use the above-mentioned target classifier to classify the image to be classified and obtain the classification result.

[0100] In this embodiment, the kernel integrated support vector machine can also be combined and improved into a classifier applicable to multi-class problems. Specifically, the one-versus-rest method can be used to extend to solve multi-class problems; the samples of a certain class are regarded as one class, and the rest are uniformly regarded as another class to obtain the same number of binary classifiers as the number of sample types, and then the samples are classified into the class with the larger test output value to obtain the classification result.

[0101] The above embodiments are only used to illustrate the design idea and characteristics of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made according to the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. An image classification method based on a kernel integrated support vector machine with a shared parameter space, characterized in that It includes the following steps: Step 1: Obtain image data to form a sample set, preprocess the sample set, and use the kernel method to obtain different individual basic kernels of the sample set; Step 2: Use constraints to integrate the losses of multiple single-core models into a unified loss, which is the integrated loss, and then construct a kernel integrated support vector machine model in the shared parameter space from multiple single-core models; The objective function of the kernel integrated support vector machine model is expressed as: S.T.1 T w = 1, w i ≥ 0, Among them, C is a parameter used to achieve the balance between the minimum empirical loss and the minimum structural loss; w i is used to control the weight of the loss in each reproducing kernel Hilbert space. w is a vector composed of weights w i . K i is the i-th kernel Gram matrix. α i and α0 are both shared parameters related to the weights of each training data sample in K i and are both column vectors of N×1. L is the number of individual base kernels. b i is the offset of K i . ξ it is the hinge loss of the t-th sample in the i-th kernel, where t = 1, 2, …, N and N is the number of samples. α 0m , α im are respectively the m-th components in the column vectors α i and α0. x t , x m are respectively the t-th sample and the m-th sample. Step 3: Input the training set into the kernel ensemble support vector machine model, and further learn the common parameter α0 and the individual specific parameter α shared by each kernel in multiple kernel spaces. i ; The designed kernel ensemble support vector machine model jointly optimizes each single kernel model by minimizing the ensemble loss; thereby obtaining a trained kernel ensemble support vector machine model based on the shared parameter space; the training process of the kernel ensemble support vector machine model is as follows: Step 1: Fix the value of w and initialize Then, the kernel integrated support vector machine model is transformed into the dual form using the Lagrange multiplier method as follows: where β is a column vector formed by Lagrange multipliers, and β t is the t-th component in the column vector β, where t = 1, 2, …, N; Y is a diagonal matrix formed by sample labels y1, y2, ..., y N , and y t is the t-th sample label, where t = 1, 2, …, N; C is the penalty coefficient Since the obtained dual form of the kernel integrated support vector machine model is the dual form of the standard SVM, the SVM solver is used to solve for the maximum value of β; Step 2: Fix the obtained maximum value of β, and update w by solving the minimization problem; The solution process is as follows: Apply the augmented Lagrangian multiplier method to the dual form of the kernel integrated support vector machine model to obtain the following formula: Among them, A i and B i are weighted basic kernel combinations, and the corresponding w j is the weight of the j-th single-core model, and the initialization parameters are: w i , λ, η i , μ; η i , μ, and λ are all Lagrange multipliers; Step 3: Take the partial derivative of the above equation with respect to w i and set it to zero to obtain w as shown in the following equation i Update formula: Step 4: Solve for λ, η i and μ in the following order, where θ is the learning rate; η i = η i + θ * w i Step 5: Based on the solved β and w i , and then complete the learning of the common parameter α0 and the single specific parameter α i , which are respectively expressed as: α i = w i Yβ where, K i is the i-th kernel Gram matrix, and K j is the j-th kernel Gram matrix, where j = 1, 2, … L, and L is the number of individual base kernels; Step 4: Set the decision condition for the iteration stop of the kernel integrated support vector machine model. When the decision condition is met, jump out of the loop; otherwise, continue to execute the loop; When the loop exits, it means that the optimal solution w has been found, thus obtaining a trained kernel integrated support vector machine model based on the shared parameter space; According to the optimal solution w, obtain where f(x t ) is the predicted label result; Step 5: Take the test set X test = [x1, x2,..., x N ∈ R N*q Input the kernel integrated support vector machine model in the trained shared parameter space to obtain the model of the final binary classifier: Y = sign(f(X test )), that is, the target classifier. Use the target classifier to classify the image to be classified and obtain the classification result.

2. The image classification method of a kernel integrated support vector machine based on a shared parameter space according to claim 1, characterized in that The decision condition for the iteration stop of the kernel integrated support vector machine model is: when the absolute value of the difference in losses before and after update, loss, is less than 1e-6 or the number of iterations is greater than 200 times, that is, the decision condition is met and the iteration stops.

3. The image classification method of a kernel integrated support vector machine based on a shared parameter space according to claim 2, wherein, Set the initial loss new = 0, and calculate Thus, we obtain: loss = |loss old - loss new |; where loss is the absolute value of the difference in losses before and after the update, loss old is the loss before the update, and loss new is the loss after the update; Recalculate according to the updated parameters: Thus, the updated loss = |loss old - loss new |, save loss old ← loss new , and increment the iteration count by 1.

4. An image classification method based on a kernel integrated support vector machine with a shared parameter space according to claim 1, characterized in that Combine the kernel integrated support vector machine and improve it into a classifier applicable to multi-class problems; Use the one-versus-rest method to extend to solve multi-class problems; Record the samples of a certain category as one class, and regard the rest as another class uniformly, to obtain the same number of binary classifiers as the number of sample types, and then classify the samples into the class with the larger test output value to obtain the classification result.

5. A method for image classification of a kernel integrated support vector machine based on a shared parameter space according to claim 1, characterized in that, In Step 2, the initial parameters are: λ = 0, η i = 0, μ = 1.

6. A method for image classification of a kernel integrated support vector machine based on a shared parameter space according to claim 1, characterized in that, Randomly select 60% of the preprocessed sample set as the training set, and the remaining 40% of the sample set as the test set.

7. An image classification method of a kernel integrated support vector machine based on a shared parameter space according to claim 6, characterized in that, The preprocessing described in Step 1 is to normalize the sample set.

Images