Image classification method based on Bayesian linear regression hyper-parameter optimization EM algorithm
Through the hyperparameter optimization EM algorithm based on Bayesian linear regression, the problem of lack of mathematical explanation and low efficiency of hyperparameter optimization in the existing technology is solved, and the image classification effect with fast convergence and high accuracy is achieved.
Patent Information
- Application Number
- CN202410956255.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-17
AI Technical Summary
The existing hyperparameter optimization methods lack rigorous mathematical explanation and are inefficient in large-scale search spaces. In particular, grid search and random search are brute-force-intensive methods.
Using a hyperparameter optimization EM algorithm based on Bayesian linear regression, the hyperparameters α, β, γ are optimized by deriving log-edge likelihood functions, and applied to the MNIST handwritten digital library for image recognition, and combining with multi-class logistic regression for classification .
The optimization effect of rapid convergence is achieved, the accuracy and generalization ability of image classification are improved, and the disadvantages of violent search by traditional methods are avoided.
Smart Images

Figure CN118968521A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image classification, and in particular to an image classification method based on a Bayesian linear regression hyperparameter optimization EM algorithm. Background Art
[0002] Parameters in machine learning can be divided into two types: model parameters are internally configurable, whose values can be estimated from data, such as the weights of deep neural networks; while hyperparameters are external, whose values cannot be estimated from data, such as learning rate, batch size, and the number of hidden layers in neural networks. Hyperparameter optimization is an important aspect of machine learning model training, and includes the process of fine-tuning the model's hyperparameters to improve its performance. Hyperparameters are parameters that are set before the learning process begins. Adjusting these hyperparameters can significantly affect the accuracy and generalization ability of the model. There are currently some hyperparameter optimization methods for machine learning as follows: Grid search: Grid search is a method of brute force search for hyperparameters with defined lower and upper bounds and a specific step size. Grid search is based on the Cartesian product of different sets of values, evaluates each configuration and returns the best performing combination. Grid search is simple to implement, but can be very inefficient for large search spaces due to its exhaustive nature. Random search: Random search randomly samples hyperparameter combinations from a predefined search space. Although it is less computationally intensive than grid search, it can usually find better hyperparameter configurations because it can explore the search space more efficiently. Genetic algorithms: Genetic algorithms are inspired by the natural selection process and evolve populations of candidate solutions (hyperparameter configurations) over multiple generations. They are suitable for exploring large search spaces and can effectively handle discrete and nonlinear search spaces. Gradient-based optimization: Gradient-based optimization treats hyperparameters as continuous variables. This approach is often used in neural network architectures and may be effective for optimizing large-scale models.
[0003] However, gradient-based methods, grid search, random search, and genetic algorithms lack strict mathematical explanations. They are more of a computational logic. At the same time, grid search and random search are brute force search methods, which are time-consuming and labor-intensive. Summary of the invention
[0004] The purpose of the present invention is to solve the problems in the prior art, and to provide an image classification method based on the hyperparameter optimization EM algorithm of Bayesian linear regression. The method has strict mathematical derivation and explanation, and the simulation results show that the method has the advantage of fast convergence speed. For the MNIST classic handwritten digit library, the regression analysis model after the hyperparameter optimization of the method of the present invention is used, and then the multi-class logistic regression is applied to classify and identify the image test set. From the effect, the classification is more accurate and has better generalization ability.
[0005] The present invention is implemented by the following technical scheme. The present invention proposes an image classification method based on the hyperparameter optimization EM algorithm of Bayesian linear regression, and the method comprises the following steps:
[0006] Step 1, obtain the logarithmic marginal likelihood function;
[0007] Step 2: Consider the parameter weight w as a latent variable and derive the EM algorithm for the hyperparameters α, β, and γ;
[0008] Step 3: Apply the hyperparameter-optimized EM algorithm to the MNIST classic handwritten digit library image recognition;
[0009] The step 3 is specifically as follows:
[0010] (1) Each of the 70,000 images of handwritten digits 0 to 9 has 28×28 pixels. The grayscale data of each image is converted into a one-dimensional vector of 28×28=784 and stored;
[0011] (2) Determine the features of each image from 0 to 9 and obtain the target variable t;
[0012] (3) Apply the EM algorithm to optimize hyperparameters on the training set;
[0013] (4) For the training set, the least squares method is used to optimize the weight parameter w and determine the value of the weight parameter w;
[0014] (5) For the regression analysis model with optimized hyperparameters and weight parameter w, multi-class logistic regression is applied to recognize and classify images of numbers 0 to 9 in the 10,000 test sets.
[0015] Furthermore, the step 1 is specifically as follows:
[0016] (1) The prior for defining weights is a Gaussian distribution;
[0017] (2) Define the linear regression model;
[0018] (3) Obtain the likelihood function;
[0019] (4) Prove that the posterior distribution of weights is also Gaussian;
[0020] (5) The marginal likelihood function, that is, the evidence function, is obtained by integrating the weight parameter using the product of the likelihood function and the Gaussian prior of the weight parameter, and the logarithmic marginal likelihood function is obtained at the same time.
[0021] Furthermore, the Gaussian prior of the weight is defined as:
[0022]
[0023] where α=(α 1 ,…,α M ) T , α i is a hyperparameter and corresponds to the parameter w i The accuracy of γ=(γ 1 ,…,γ M ) T , γ i is a hyperparameter and corresponds to the parameter w i The mean of
[0024] The linear regression model is defined as:
[0025] The linear regression model is defined as a linear combination of nonlinear functions of the input variables in the form
[0026]
[0027] where φ i (x) is the basis function, w 0 is called the bias parameter, defining φ 0 (x) = 1 and we get
[0028]
[0029] Where w=(w 0 ,…,w M-1 ) T ,φ(x)=(φ 0 (x),…,φ M-1 (x) T .
[0030] Furthermore, the likelihood function derivation process is:
[0031] The target variable t is defined by a deterministic function y(x,w) with Gaussian noise
[0032] t=y(x,w)+ε, (4)
[0033] where ε is a Gaussian random variable with zero mean and precision β, thus we have
[0034] p(t|x,w,β)=N(t|y(x,w),β -1 ) (5)
[0035] An input data set X = (x 1 ,…,x N ) and its corresponding target vector t=(t 1 ,…,t N ), where the data points are independently sampled from the Gaussian distribution (5), from which we can obtain the following likelihood function
[0036]
[0037] Furthermore, the proof process is:
[0038] Since (6) is the exponential form of the quadratic function w, the corresponding conjugate prior is defined as
[0039] p(w)=N(w|m 0 ,S 0 ), (7)
[0040] From (6) and (7), we get the following posterior distribution:
[0041] p(w|t,X,β)=N(w|m N ,S N ), (8)
[0042] here
[0043]
[0044] where Φ is an N×M matrix whose elements are defined as Φ nj =φ j (x n ),satisfy
[0045]
[0046] From (6), we can get
[0047]
[0048]
[0049] From (1), we can get
[0050]
[0051] The posterior distribution of weights is also Gaussian and has the following form
[0052] p(w|t,X,α,β,γ)=N(w|m,Σ), (14)
[0053] Here the mean and covariance are
[0054] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (15)
[0055] Σ=(A+βΦ T Φ) -1 , (16)
[0056] Where A = diag (α i ).
[0057] Furthermore, the weight parameter is integrated by multiplying the likelihood function by the Gaussian prior of the weight parameter to obtain the marginal likelihood function, that is, the evidence function, and the specific derivation process of the logarithmic marginal likelihood function is:
[0058] First, integrate the weight parameters to obtain the marginal likelihood function, also known as the evidence function
[0059] p(t|X,α,β,γ)=∫p(t|X,w,β)p(w|α,γ)dw, (17)
[0060] Substituting (12) and (13) into (17), we can obtain
[0061]
[0062] Here is defined
[0063]
[0064] The gradient of E(w)▽E(w) is
[0065] ▽E(w)=(A+βΦ T Φ)w-(Aγ+βΦ T t), (20)
[0066] ▽▽E(w)=A+βΦ T Φ=Σ -1 (twenty one)
[0067] let
[0068] ▽E(w)=(βΦ T Φ+A)w-(A+βΦ T t)=0, (22)
[0069] Using (15) and (22) at the same time, we can get the stationary point of E(w) as
[0070] w=(A+βΦ T Φ) -1 (Aγ+βΦ T t)=m (23)
[0071] Using (21) and (23), we can get
[0072]
[0073] here
[0074]
[0075] Substituting (24) into (18) we get
[0076]
[0077]
[0078] here
[0079] From (26), the log marginal likelihood function is
[0080]
[0081] Furthermore, the step 2 is specifically as follows:
[0082] ①The joint distribution p(t,w|X,α,β,γ) on t and w is equal to the product of the likelihood function and the prior probability of weight w, that is,
[0083] p(t,w|X,α,β,γ)=p(t|X,w,β)p(w|α,γ);
[0084] ② Treat the parameter weight w as a hidden variable, integrate the joint distribution p(t,w|X,α,β,γ) on t and w with respect to w, and then obtain the logarithmic marginal likelihood function
[0085] L=ln p(t|X,α,β,γ)=ln∫p(t,w|X,α,β,γ)dw;
[0086] ③ Maximization of log-likelihood function;
[0087] ④ Obtain the E step and M step of the EM algorithm and the simulation results.
[0088] Furthermore, the derivation process of step ③ is:
[0089] Define a variational probability distribution q(w) with respect to the weights and use Jensen’s inequality to obtain a lower bound on L
[0090]
[0091] Here θ represents the remaining parameters. The EM algorithm maximizes the log-marginal likelihood L by iteratively maximizing the lower bound. In the expectation step E, the hyperparameters α, β, and γ are fixed, and F is maximized with respect to the variational distribution q(w). In the maximization step M, the hyperparameters α, β, and γ are maximized with respect to the variational distribution q(w). The E step is performed by rewriting the lower bound as F to obtain
[0092] F=L+KL(q(w)||p(w|t,X,α,β,γ)), (27)
[0093] Here KL(q(w)||p(w|t,X,α,β,γ)) is the Kullback-Leibler (KL) divergence between the variational distribution q(w) and the posterior distribution of the weights. The E step corresponds to the posterior distribution that makes q(w) equal to w, that is, F = L and q(w) = p(w|t,X,α,β,γ). Since the posterior distribution is Gaussian, the E step simplifies to computing its mean m and covariance Σ, which is defined by (15) and (16);
[0094] To implement the M step, rewrite F as
[0095] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w)), (28)
[0096] Here H(q(w)) is the information entropy of q(w), and H(q(w)) is a quantity that is independent of α, β, and γ. Therefore, for the M step of the EM algorithm, it is necessary to maximize the expected value of the logarithmic joint distribution of t and w in q(w) for α, β, and γ.
[0097] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w))
[0098] =∫q(w)(ln p(w|α,γ)+ln p(t|X,w,β))dw-H(q(w))
[0099] =∫q(w)ln p(w|α,γ)dw+∫q(w)ln p(t|X,w,β)dw-H(q(w)) (29)
[0100] Calculating the first term of (29), we can get
[0101]
[0102] Here ∫q(w)dw=1
[0103] The third integral of (30) is
[0104] ∫q(w)(w-γ) T A(w-γ)dw
[0105] =∫q(w)Tr(A(w-γ)(w-γ) T )dw
[0106] =∫q(w)Tr(Aww T -wγ T -γw T +γγ T )dw
[0107] =∫q(w)(Tr(Aww T )-2Tr(wγ T )+Tr(γγ T ))dw
[0108] =∫q(w)Tr(Aww T )dw-2∫q(w)Tr(wγ T )dw+∫q(w)Tr(γγ T )dw
[0109] =Tr(A∫q(w)ww T dw)-2Tr((∫q(w)wdw)γ T )+Tr(∫q(w)γγ T dw)
[0110] =Tr(AE q (ww T ))-2Tr(E q (w)γ T )+Tr(γγ T )
[0111] =Tr(A(mm T +Σ))-2Tr(mγ T )+Tr(γγ T )
[0112] =Tr(Amm T )+Tr(AΣ)-2Tr(mγ T )+Tr(γγ T )
[0113] =m T Am+Tr(AΣ)-2γ T m+γ T γ, (31)Here E q (·) is the mathematical expectation of the random variable on the distribution q(w) and E q (ww T )=mm T +Σ, Tr(·) is the trace of the matrix, and the mean m and covariance Σ are defined by (15) and (16),
[0114] From (30) and (31), we can get
[0115]
[0116] Calculate the second term of (29)
[0117]
[0118] (33).
[0119] ∫q(w)(t) T t-2w T F T t+w T F T Φw)dw
[0120] =∫q(w)t T tdw-2∫q(w)w T F T tdw+∫q(w)w T F T Φwdw
[0121] =t T t-2∫q(w)w T F T tdw+vq(w)Tr(Φww T F T )dw
[0122] =t T t-2∫q(w)w T F T tdw+Tr∫q(w)Φww T F T dw
[0123] =t T t-2(∫q(w)w T dw)Φ T t+Tr(Φ(∫q(w)ww T dw)Φ T )
[0124] =t T t-2E q (w T )F T t+Tr(ΦE q (ww T )F T )
[0125] =t T t-2m T F T t+Tr(Φ(mm T +S)F T )
[0126] =t T t-2m T F T t+Tr(Φmm T F T +FSF T )
[0127] =t T t-2m T Φ T t+Tr(Φmm T Φ T )+Tr(ΦΣΦ T )
[0128] =t T t-2m T Φ T t+m T Φ T Φm+Tr(ΦΣΦ T )
[0129] =(t-Φm) T (t-Φm)+Tr(ΦΣΦ T )
[0130] =||t-Φm|| 2 +Tr(ΦΣΦ T )
[0131] =||t-Φm|| 2 +Tr(ΣΦ T Φ)
[0132] =||t-Φm 2 +Tr(Φ T ΦΣ), (34)
[0133] Here E q (ww T )=mm T +Σ.
[0134] Using (33) and (34), we can get
[0135]
[0136] From (32) and (35), we can get
[0137]
[0138] From this we can get (36) for α i , the stationary points of β and γ satisfy
[0139]
[0140] Therefore, the update equation for the hyperparameters is as follows
[0141]
[0142] γ new =m, (42)
[0143] Here m i is the i-th element of the posterior distribution mean m defined by (15), Σ ii is the i-th element on the main diagonal of the posterior covariance Σ defined by (16).
[0144] Furthermore, the step ④ is specifically as follows:
[0145] (1) Randomly generate the values of hyperparameters α, β, γ and the data set X = {x 1 ,…,x N};
[0146] (2) According to (11), select a suitable linear basis function φ(x n ) to construct an N×M matrix Φ;
[0147] (3) Through (1), randomly sample the weight parameter w;
[0148] (4) Generate an n-dimensional vector ε randomly sampled from the Gaussian distribution N(0,β), and then generate t=(t 1 ,…,t N );
[0149] (5) Step E: Estimate the mean and covariance of the posterior distribution defined by (14) using the current hyperparameter values generated by step (1). Estimate the mean m and covariance Σ
[0150] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (43)
[0151] Σ=(A+βΦ T Φ) -1 (44)
[0152] (6) Step M: Use the mean m and covariance Σ obtained in step (5) and the following update equation to estimate the hyperparameters
[0153]
[0154] γ new =m (47)
[0155] (7) Estimation of log-likelihood
[0156]
[0157] Then check the convergence of hyperparameters or log-likelihood, and return to step (5) if the convergence criteria are not met.
[0158] The present invention has the following beneficial effects:
[0159] (1) The present invention uses Bayesian theory to derive the EM algorithm, which has a rigorous mathematical interpretation;
[0160] (2) Through simulation results, the EM algorithm has the advantage of fast convergence speed and avoids the shortcomings of some classic algorithms such as brute force search;
[0161] (3) For the MNIST classic handwritten digit library, the regression analysis model after hyperparameter optimization by the method described in the present invention is used, and then multi-class logistic regression is applied to classify and recognize the image test set. From the effect point of view, the classification is more accurate and has better generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0162] Figure 1 It is the convergence diagram of EM algorithm;
[0163] Figure 2 It is the flow chart of EM algorithm;
[0164] Figure 3 This is the flow chart of the image classification algorithm. DETAILED DESCRIPTION
[0165] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0166] See also Figure 1-Figure 3 The present invention proposes an image classification method based on a hyperparameter optimized EM algorithm of Bayesian linear regression, the method comprising the following steps:
[0167] Step 1, obtain the logarithmic marginal likelihood function;
[0168] The step 1 is specifically as follows:
[0169] (1) The prior for defining weights is a Gaussian distribution;
[0170] (2) Define the linear regression model;
[0171] (3) Obtain the likelihood function;
[0172] (4) Prove that the posterior distribution of weights is also Gaussian;
[0173] (5) The marginal likelihood function, that is, the evidence function, is obtained by integrating the weight parameter using the product of the likelihood function and the Gaussian prior of the weight parameter, and the logarithmic marginal likelihood function is obtained at the same time.
[0174] The Gaussian prior for the weights is defined as:
[0175]
[0176] where α=(α 1 ,…,α M ) T , α i is a hyperparameter and corresponds to the parameter w i The accuracy of 1 ,…,γ M ) T , γ i is a hyperparameter and corresponds to the parameter w i The mean of
[0177] The linear regression model is defined as:
[0178] The linear regression model is defined as a linear combination of nonlinear functions of the input variables in the form
[0179]
[0180] where φ i (x) is the basis function, w 0 is called the bias parameter, defining φ 0 (x) = 1 and we get
[0181]
[0182] Where w=(w 0 ,…,w M-1 ) T ,φ(x)=(φ 0 (x),…,φ M-1 (x)) T .
[0183] The likelihood function derivation process is:
[0184] The target variable t is defined by a deterministic function y(x,w) with Gaussian noise
[0185] t=y(x,w)+ε, (51)
[0186] where ε is a Gaussian random variable with zero mean and precision (inverse of variance) β, thus we have
[0187] p(t|x,w,β)=N(t|y(x,w),β -1 ) (52)
[0188] An input data set X = (x 1,…,x N ) and its corresponding target vector t=(t 1 ,…,t N ), where the data points are independently sampled from the Gaussian distribution (5), from which we can obtain the following likelihood function
[0189]
[0190] The proof process is:
[0191] Since (6) is the exponential form of the quadratic function w, the corresponding conjugate prior is defined as
[0192] p(w)=N(w|m 0 ,S 0 ), (54)
[0193] From (6) and (7), we get the following posterior distribution:
[0194] p(w|t,X,β)=N(w|m N ,S N ), (55)
[0195] here
[0196]
[0197] where Φ is an N×M matrix whose elements are defined as Φ nj =φ j (x n ),satisfy
[0198]
[0199] From (6), we can get
[0200]
[0201] From (1), we can get
[0202]
[0203]
[0204] The posterior distribution of weights is also Gaussian and has the following form
[0205] p(w|t,X,α,β,γ)=N(w|m,Σ), (61)
[0206] Here the mean and covariance are
[0207] m=(A+βΦ T Φ) -1 (Aγ+βΦT t), (62)
[0208] Σ=(A+βΦ T Φ) -1 , (63)
[0209] Where A = diag (α i ).
[0210] The weight parameter is integrated by multiplying the likelihood function by the Gaussian prior of the weight parameter to obtain the marginal likelihood function, that is, the evidence function, and the specific derivation process of the logarithmic marginal likelihood function is:
[0211] First, integrate the weight parameters to obtain the marginal likelihood function, also known as the evidence function
[0212] p(t|X,α,β,γ)=∫p(t|X,w,β)p(w|α,γ)dw, (64)
[0213] Substituting (12) and (13) into (17), we can obtain
[0214]
[0215] Here is defined
[0216]
[0217] The gradient of E(w)▽E(w) is
[0218] ▽E(w)=(A+βΦ T Φ)w-(Aγ+βΦ T t), (67)
[0219] ▽▽E(w)=A+βΦ T Φ=Σ -1 (68)
[0220] let
[0221] ▽E(w)=(βΦ T Φ+A)w-(A+βΦ T t)=0, (69)
[0222] Using (15) and (22) at the same time, we can get the stationary point of E(w) as
[0223] w=(A+βΦ T Φ) -1 (Aγ+βΦ T t)=m (70)
[0224] Using (21) and (23), we can get
[0225]
[0226] here
[0227]
[0228] Substituting (24) into (18) we get
[0229]
[0230] here
[0231] From (26), the log-marginal likelihood function is
[0232]
[0233] Step 2: Consider the parameter weight w as a latent variable and derive the EM algorithm for the hyperparameters α, β, and γ;
[0234] The step 2 is specifically as follows:
[0235] ①The joint distribution p(t,w|X,α,β,γ) on t and w is equal to the product of the likelihood function and the prior probability of weight w, that is,
[0236] p(t,w|X,α,β,γ)=p(t|X,w,β)p(w|α,γ);
[0237] ② Treat the parameter weight w as a hidden variable, integrate the joint distribution p(t,w|X,α,β,γ) on t and w with respect to w, and then obtain the logarithmic marginal likelihood function
[0238] L=ln p(t|X,α,β,γ)=ln∫p(t,w|X,α,β,γ)dw;
[0239] ③Maximize the log-likelihood function;
[0240] ④ Obtain the E step and M step of the EM algorithm and the simulation results.
[0241] The derivation process of step ③ is:
[0242] Define a variational probability distribution q(w) with respect to the weights and use Jensen’s inequality to obtain a lower bound on L
[0243]
[0244] Here θ represents the remaining parameters. The EM algorithm maximizes the log-marginal likelihood L by iteratively maximizing the lower bound. In the expectation step E, the hyperparameters α, β, and γ are fixed, and F is maximized with respect to the variational distribution q(w). In the maximization step M, the hyperparameters α, β, and γ are maximized with respect to the variational distribution q(w). The E step is performed by rewriting the lower bound as F to obtain
[0245] F=L+KL(q(w)||p(w|t,X,α,β,γ)), (74)
[0246] Here KL(q(w)||p(w|t,X,α,β,γ)) is the Kullback-Leibler (KL) divergence between the variational distribution q(w) and the posterior distribution of the weights. The KL divergence is always a positive number and is zero only when the two distributions are exactly the same. The E step corresponds to the posterior distribution that makes q(w) equal to w, that is, F = L and q(w) = p(w|t,X,α,β,γ). Since the posterior distribution is a Gaussian distribution, the E step simplifies to calculating its mean m and covariance Σ, which is defined by (15) and (16);
[0247] To implement the M step, rewrite F as
[0248] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w)), (75)
[0249] Here H(q(w)) is the information entropy of q(w), and H(q(w)) is a quantity that is independent of α, β, and γ. Therefore, for the M step of the EM algorithm, it is necessary to maximize the expected value of the logarithmic joint distribution of t and w in q(w) for α, β, and γ.
[0250] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w))
[0251] =∫q(w)(ln p(w|α,γ)+ln p(t|X,w,β))dw-H(q(w))
[0252] =∫q(w)ln p(w|α,γ)dw+∫q(w)ln p(t|X,w,β)dw-H(q(w)) (76)
[0253] Calculating the first term of (29), we can get
[0254]
[0255] Here ∫q(w)dw=1
[0256] The third integral of (30) is
[0257] ∫q(w)(w - γ) T A(w - γ)dw
[0258] =∫q(w)Tr(A(w - γ)(w - γ) T )dw
[0259] =∫q(w)Tr(Aww T - wγ T - γw T + γγ T )dw
[0260] =∫q(w)(Tr(Aww T ) - 2Tr(wγ T ) + Tr(γγ T ))dw
[0261] =∫q(w)Tr(Aww T )dw - 2∫q(w)Tr(wγ T )dw + ∫q(w)Tr(γγ T )dw
[0262] =Tr(A∫q(w)ww T dw) - 2Tr((∫q(w)wdw)γ T ) + Tr(∫q(w)γγ T dw)
[0263] =Tr(AE q (ww T )) - 2Tr(E q (w)γ T ) + Tr(γγ T )
[0264] =Tr(A(mm T + Σ)) - 2Tr(mγ T ) + Tr(γγ T )
[0265] =Tr(Amm T ) + Tr(AΣ) - 2Tr(mγ T ) + Tr(γγ T )
[0266] =m T Am + Tr(AΣ) - 2γ T m + γ T γ, (78) Here E q (·) is the mathematical expectation of the random variable over the distribution q(w) and E q (ww T )=mmT +Σ, Tr(·) is the trace of the matrix, and the mean m and covariance Σ are defined by (15) and (16),
[0267] From (30) and (31), we can get
[0268]
[0269] Calculate the second term of (29)
[0270]
[0271] The third integral of (33) gives
[0272] ∫q(w)(t T t-2w T Φ T t+w T Φ T Φw)dw
[0273] =∫q(w)t T tdw-2∫q(w)w T Φ T tdw+∫q(w)w T Φ T Φwdw
[0274] =t T t-2∫q(w)w T Φ T tdw+∫q(w)Tr(Φww T Φ T )dw
[0275] =t T t-2∫q(w)w T Φ T tdw+Tr∫q(w)Φww T Φ T dw
[0276] =t T t-2(∫q(w)w T dw)Φ T t+Tr(Φ(∫q(w)ww T dw)Φ T )
[0277] =t T t-2E q (w T )Φ T t+Tr(ΦE q (ww T )Φ T )
[0278] = t T t - 2m T Φ T t + Tr(Φ(mm T + Σ)Φ T )
[0279] = t T t - 2m T Φ T t + Tr(Φmm T Φ T + ΦΣΦ T )
[0280] = t T t - 2m T Φ T t + Tr(Φmm T Φ T ) + Tr(ΦΣΦ T )
[0281] = t T t - 2m T Φ T t + m T Φ T Φm + Tr(ΦΣΦ T )
[0282] = (t - Φm) T (t - Φm) + Tr(ΦΣΦ T )
[0283] = ||t - Φm|| 2 + Tr(ΦΣΦ T )
[0284] = ||t - Φm|| 2 + Tr(ΣΦ T Φ)
[0285] = ||t - Φm|| 2 + Tr(Φ T ΦΣ), (81)
[0286] Here E q (ww T ) = mm T + Σ。
[0287] Using (33) and (34), we can obtain
[0288]
[0289] From (32) and (35), we can obtain
[0290]
[0291] From this we can get (36) for α i , the stationary points of β and γ satisfy
[0292]
[0293] Therefore, the update equation for the hyperparameters is as follows
[0294]
[0295] γ new =m, (89)
[0296] Here m i is the i-th element of the posterior distribution mean m defined by (15), Σ ii is the i-th element on the main diagonal of the posterior covariance Σ defined by (16).
[0297] The step ④ is specifically as follows:
[0298] (1) Randomly generate the values of hyperparameters α, β, γ and the data set X = {x 1 ,…,x N};
[0299] (2) According to (11), select a suitable linear basis function φ(x n ) to construct an N×M matrix Φ;
[0300] (3) Through (1), randomly sample the weight parameter w;
[0301] (4) Generate an n-dimensional vector ε randomly sampled from the Gaussian distribution N(0,β), and then generate t=(t 1 ,…,t N );
[0302] (5) Step E: Estimate the mean and covariance of the posterior distribution defined by (14) using the current hyperparameter values generated by step (1). Estimate the mean m and covariance Σ
[0303] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (90)
[0304] Σ=(A+βΦ T Φ) -1 (91)
[0305] (6) Step M: Use the mean m and covariance Σ obtained in step (5) and the following update equation to estimate the hyperparameters
[0306]
[0307]
[0308] γ new =m (94)
[0309] (7) Estimation of log-likelihood
[0310]
[0311] Then check the convergence of hyperparameters or log-likelihood, and return to step (5) if the convergence criteria are not met.
[0312] The present invention applies the obtained EM algorithm to optimize the values of hyperparameters α, β and γ. Here, the linear basis function φ(x) is a Gaussian basis function, and it is difficult to judge the convergence of the hyperparameters α, β and γ. Therefore, it is reasonable to judge the convergence of the log-likelihood instead, because the convergence of the hyperparameters α, β and γ means that the log-likelihood remains unchanged. In other words, the convergence of the log-likelihood is much easier to implement in programming. Figure 1 It can be concluded that through rigorous mathematical and statistical interpretation, the obtained EM algorithm has the advantage of fast convergence.
[0313] Step 3: Apply the hyperparameter-optimized EM algorithm to the MNIST classic handwritten digit library image recognition;
[0314] The step 3 is specifically as follows:
[0315] (1) Each of the 70,000 images of handwritten digits 0 to 9 (including 60,000 training images and 10,000 test images) has 28 × 28 pixels. The grayscale data of each image is converted into a one-dimensional vector of 28 × 28 = 784 and stored;
[0316] (2) Determine the features of each image from 0 to 9 and obtain the target variable t;
[0317] (3) Apply the EM algorithm to optimize hyperparameters on the training set;
[0318] The specific steps of step (3) are as follows:
[0319] 1) For 60,000 training set images, construct a dataset X = {x 1 ,…,x 60000} and the corresponding target set t=(t 1 ,…,t N );
[0320] 2) According to formula (11), select the appropriate linear basis function φ(x n ) to construct the matrix Φ,
[0321] 3) According to formula (1), the random sampling weight parameter w is
[0322] 4) Step E: Estimate the mean and covariance of the posterior distribution defined by formula (14) using the current hyperparameter values generated by step 1) to estimate the mean m and covariance Σ
[0323] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (95)
[0324] Σ=(A+βΦ T Φ) -1 . (96)
[0325] 5) Step M: Use the mean m and covariance Σ obtained in step 4) and the following update equation to estimate the hyperparameters
[0326]
[0327] γ new =m (99)
[0328] 6) Estimating the log-likelihood
[0329]
[0330] Then check the convergence of hyperparameters or log-likelihood. If the convergence criteria are not met, return to step 4).
[0331] (4) For the training set, the least squares method is used to optimize the weight parameter w and determine the value of the weight parameter w;
[0332] (5) For the regression analysis model with optimized hyperparameters and weight parameter w, multi-class logistic regression is applied to recognize and classify images of numbers 0 to 9 in the 10,000 test sets.
[0333] The above only expresses the preferred implementation mode of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.
Claims
1. An image classification method based on the hyperparameter optimization EM algorithm of Bayesian linear regression, characterized by: The method comprises the following steps: Step 1, obtain the logarithmic marginal likelihood function; Step 2: Consider the parameter weight w as a latent variable and derive the EM algorithm for the hyperparameters α, β, and γ; Step 3: Apply the hyperparameter-optimized EM algorithm to the MNIST classic handwritten digit library image recognition; The step 3 is specifically as follows: (1) Each of the 70,000 images of handwritten digits 0 to 9 has 28×28 pixels. The grayscale data of each image is converted into a one-dimensional vector of 28×28=784 and stored; (2) Determine the features of each image from 0 to 9 and obtain the target variable t; (3) Apply the EM algorithm to optimize hyperparameters on the training set; (4) For the training set, the least squares method is used to optimize the weight parameter w and determine the value of the weight parameter w; (5) For the regression analysis model with optimized hyperparameters and weight parameter w, multi-class logistic regression is applied to recognize and classify images of numbers 0 to 9 in the 10,000 test sets.
2. The method according to claim 1, characterized in that The step 1 is specifically as follows: (1) The prior for defining weights is a Gaussian distribution; (2) Define the linear regression model; (3) Obtain the likelihood function; (4) Prove that the posterior distribution of weights is also Gaussian; (5) The marginal likelihood function, that is, the evidence function, is obtained by integrating the weight parameter using the product of the likelihood function and the Gaussian prior of the weight parameter, and the logarithmic marginal likelihood function is obtained at the same time.
3. The method according to claim 2, characterized in that The Gaussian prior for the weights is defined as: where α=(α1,…,α M ) T , α i is a hyperparameter and corresponds to the parameter w i The accuracy of γ=(γ1,…,γ M ) T , γ i is a hyperparameter and corresponds to the parameter w i The mean of . The linear regression model is defined as: The linear regression model is defined as a linear combination of nonlinear functions of the input variables in the form where φ i (x) is the basis function, w0 is called the deviation parameter, and we define φ0(x) = 1 and get where \(w=(w_0,\ldots,w M-1 ) T \), and \(\varphi(x)=(\varphi_0(x),\ldots,\varphi M-1 (x)) T \).
4. The method according to claim 3, characterized in that The likelihood function derivation process is: The target variable t is defined by a deterministic function y(x,w) with Gaussian noise t=y(x,w)+ε, (4) where ε is a Gaussian random variable with zero mean and precision β, thus we have p(t|x,w,β)=N(t|y(x,w),β -1 ) (5) An input data set X = (x1,…,x N ) and its corresponding target vector t=(t1,…,t N ), where the data points are independently sampled from the Gaussian distribution (5), from which we can obtain the following likelihood function 5. The method according to claim 4, characterized in that The proof process is: Since (6) is the exponential form of the quadratic function w, the corresponding conjugate prior is defined as p(w)=N(w|m0,S0), (7) From (6) and (7), we get the following posterior distribution: p(w|t,X,β)=N(w|m N ,S N ), (8) here where Φ is an N×M matrix whose elements are defined as Φ nj =φ j (x n ),satisfy From (6), we can get From (1), we can get The posterior distribution of weights is also Gaussian and has the following form p(w|t,X,α,β,γ)=N(w|m,Σ), (14) Here the mean and covariance are m=(A+βΦ T F) -1 (Aγ+βΦ T t), (15) Σ=(A+βΦ T F) -1 , (16) Where A = diag(α i ).
6. The method according to claim 5, characterized in that The weight parameter is integrated by multiplying the likelihood function by the Gaussian prior of the weight parameter to obtain the marginal likelihood function, that is, the evidence function, and the specific derivation process of the logarithmic marginal likelihood function is: First, integrate the weight parameters to obtain the marginal likelihood function, also known as the evidence function p(t|X,α,β,γ)=∫p(t|X,w,β)p(w|α,γ)dw, (17) Substituting (12) and (13) into (17), we can obtain Here is defined Gradient of E(w) for let Using (15) and (22) at the same time, we can get the stationary point of E(w) as w=(A+βΦ T F) -1 (Aγ+βΦ T t)=m (23) Using (21) and (23), we can get here Substituting (24) into (18) we get here From (26), the log-marginal likelihood function is 7. The method according to claim 6, characterized in that The step 2 is specifically as follows: ① t The joint distribution p(t,w|X,α,β,γ) on w is equal to the product of the likelihood function and the prior probability of weight w, that is, p(t,w|X,α,β,γ)=p(t|X,w,β)p(w|α,γ); ② Treat the parameter weight w as a hidden variable, integrate the joint distribution p(t,w|X,α,β,γ) on t and w with respect to w, and then obtain the logarithmic marginal likelihood function L=lnp(t|X,α,β,γ)=ln∫p(t,w|X,α,β,γ)dw; ③Maximize the log-likelihood function; ④ Obtain the E step and M step of the EM algorithm and the simulation results.
8. The method according to claim 7, characterized in that The derivation process of step ③ is: Define a variational probability distribution q(w) with respect to the weights and use Jensen’s inequality to obtain a lower bound on L Here θ represents the remaining parameters. The EM algorithm maximizes the log-marginal likelihood L by iteratively maximizing the lower bound. In the expectation step E, the hyperparameters α, β, and γ are fixed, and F is maximized with respect to the variational distribution q(w). In the maximization step M, the hyperparameters α, β, and γ are maximized with respect to the variational distribution q(w). The E step is performed by rewriting the lower bound as F to obtain F=L+KL(q(w)||p(w|t,X,α,β,γ)), (27) Here KL(q(w)||p(w|t,X,α,β,γ)) is the Kullback-Leibler (KL) divergence between the variational distribution q(w) and the posterior distribution of the weights. The E step corresponds to the posterior distribution that makes q(w) equal to w, that is, F = L and q(w) = p(w|t,X,α,β,γ). Since the posterior distribution is Gaussian, the E step simplifies to computing its mean m and covariance Σ, which is defined by (15) and (16); To implement the M step, rewrite F as F=∫q(w)lnp(t,w|X,α,β,γ)dw-H(q(w)), (28) Here H(q(w)) is the information entropy of q(w), and H(q(w)) is a quantity that is independent of α, β, and γ. Therefore, for the M step of the EM algorithm, it is necessary to maximize the expected value of the logarithmic joint distribution of t and w in q(w) for α, β, and γ. F=∫q(w)lnp(t,w|X,α,β,γ)dw-H(q(w)) =∫q(w)(lnp(w|α,γ)+lnp(t|X,w,β))dw-H(q(w)) =∫q(w)lnp(w|α,γ)dw+∫q(w)lnp(t|X,w,β)dw-H(q(w)) (29) Calculate the first term of (29) Here ∫q(w)dw=1. The third integral of (30) yields ∫q(w)(w-γ) T A(w-γ)dw =∫q(w)Tr(A(w-γ)(w-γ) T )dw =∫q(w)Tr(Aww T -wγ T -γw T +gg T )dw =∫q(w)(Tr(Aww T )-2Tr(inγ T )+Tr(γγ T ))dw =∫q(w)Tr(Aww T )dw-2∫q(w)Tr(wγ T )dw+∫q(w)Tr(γγ T )dw =Tr(A∫q(w)ww T dw)-2Tr((∫q(w)wdw)γ T )+Tr(∫q(w)γγ T dw) =Tr(AE q (ww T ))-2Tr(E q (in)γ T )+Tr(γγ T ) =Tr(A(mm T +Σ))-2Tr(mγ T )+Tr(γγ T ) =Tr(Amm T )+Tr(AΣ)-2Tr(mγ T )+Tr(γγ T ) =m T Am+Tr(AΣ)-2γ T m+γ T γ, (31)Here E q (·) is the mathematical expectation of the random variable on the distribution q(w) and E q (ww T )=mm T +Σ, Tr(·) is the trace of the matrix, and the mean m and covariance Σ are defined by (15) and (16). From (30) and (31), we can get Calculate the second term of (29) The third integral of (33) gives ∫q(w)(t) T t-2w T F T t+w T F T Φw)dw =∫q(w)t T tdw-2∫q(w)w T Φ T tdw+∫q(w)w T Φ T Φwdw =t T t-2∫q(w)w T F T tdw+∫q(w)Tr(Φww T F T )dw =t T t-2∫q(w)w T F T tdw+Tr∫q(w)Φww T F T dw =t T t-2(∫q(w)w T dw)Φ T t+Tr(Φ(∫q(w)ww T dw)Φ T ) =t T t-2E q (w T )Φ T t+Tr(ΦE q (ww T )Φ T ) =t T t-2m T F T t+Tr(Φ(mm T +S)F T ) =t T t-2m T Φ T t+Tr(Φmm T Φ T +ΦΣΦ T ) =t T t-2m T Φ T t+Tr(Φmm T Φ T )+Tr(ΦΣΦ T ) =t T t-2m T Φ T t+m T Φ T Φm+Tr(ΦΣΦ T ) =(t-Φm) T (t-Φm)+Tr(ΦΣΦ T ) =||t-Φm|| 2 +Tr(ΦΣΦ T ) =||t-Φm|| 2 +Tr(ΣΦ T F) =||t-Φm|| 2 +Tr(Φ T FS), (34) Here E q (ww T )=mm T +Σ; Using (33) and (34), we can get From (32) and (35), we can get F=∫q(w)lnp(t,w|X,α,β)dw-H(q(w)) From this we can get (36) for α i , the stationary points of β and γ satisfy Therefore, the update equation for the hyperparameters is as follows Here m i is the i-th element of the posterior distribution mean m defined by (15), Σ ii is the i-th element on the main diagonal of the posterior covariance Σ defined by (16).
9. The method according to claim 8, characterized in that The step ④ is specifically as follows: (1) Randomly generate the values of hyperparameters α, β, γ and the data set X = {x1,…,x N }; (2) According to (11), select a suitable linear basis function φ(x n ) to construct an N×M matrix Φ; (3) Through (1), randomly sample the weight parameter w; (4) Generate an n-dimensional vector ε randomly sampled from the Gaussian distribution N(0,β), and then generate t=(t1,…,t N ); (5) Step E: Estimate the mean and covariance of the posterior distribution defined by (14) using the current hyperparameter values generated by step (1). Estimate the mean m and covariance Σ m=(A+βΦ T F) -1 (Aγ+βΦ T t), (43) Σ=(A+βΦ T F) -1 (44) (6) Step M: Use the mean m and covariance Σ obtained in step (5) and the following update equation to estimate the hyperparameters c new =m (47) (7) Estimation of log-likelihood Then check the convergence of hyperparameters or log-likelihood, and return to step (5) if the convergence criteria are not met.
Citation Information
Patent Citations
Network space-time monitoring method based on group sparse Bayesian learning
CN106845511A
Variational expectation maximization routing algorithm based on a capsule network
CN109583563A
Optical remote sensing image target detection method based on Bayesian transfer learning
CN110245587A
High-energy flash image deblurring method containing noise blurring
CN111028168A
Hyper-parameter image segmentation method based on Bayesian deep learning
CN111652317A
