An image classification method based on EM algorithm of hyperparameter optimization of Bayesian linear regression

Abstract

The application provides an image classification method based on a Bayesian linear regression EM algorithm of super parameter optimization, which comprises the following steps: step 1, obtaining a logarithmic edge likelihood function; step 2, regarding the parameter weight w as a hidden variable, and deducing an EM algorithm of super parameters alpha, beta and gamma; and step 3, applying the EM algorithm of super parameter optimization to MNIST classic handwritten digital library image recognition. For the MNIST classic handwritten digital library, the regression analysis model after the super parameter optimization of the method is applied to the classification and recognition of an image test set by using multiclass logistic regression, and the classification is more accurate and has better generalization ability.

Description

Technical Field

[0001] This invention relates to the field of image classification technology, and in particular to an image classification method based on the hyperparameter optimization EM algorithm of Bayesian linear regression. Background Technology

[0002] In machine learning, parameters can be categorized into two types: model parameters are internally configurable, and their values ​​can be estimated from the data, such as the weights of a deep neural network; while hyperparameters are external, and their values ​​cannot be estimated from the data, such as the learning rate, batch size, and the number of hidden layers in a neural network. Hyperparameter optimization is a crucial aspect of machine learning model training, encompassing the process of fine-tuning the model's hyperparameters to improve its performance. Hyperparameters are parameters set before the learning process begins. Adjusting these hyperparameters can significantly impact the model's accuracy and generalization ability. Currently, several hyperparameter optimization methods exist in machine learning, including: Grid Search: Grid search is a brute-force method for searching hyperparameters, with defined lower and upper bounds and a specific step size. It evaluates each configuration based on the Cartesian product of different sets of values ​​and returns the best-performing combination. While simple to implement, grid search can be very inefficient for large search spaces due to its exhaustive nature. Random search: Random search randomly samples hyperparameter combinations from a predefined search space. Although computationally less than grid search, it can often find better hyperparameter configurations because it can explore the search space more efficiently. Genetic algorithm: Inspired by the process of natural selection, genetic algorithms evolve a population of candidate solutions (hyperparameter configurations) over multiple generations. They are suitable for exploring large search spaces and can effectively handle discrete and nonlinear search spaces. Gradient-based optimization: Gradient-based optimization treats hyperparameters as continuous variables. This method is often used in neural network architectures and can be very effective for optimizing large-scale models.

[0003] However, gradient-based methods, grid search, random search, and genetic algorithms all lack rigorous mathematical interpretation; they are more of a computational logic. Furthermore, grid search and random search are brute-force search methods, which are time-consuming and laborious. Summary of the Invention

[0004] The purpose of this invention is to address the problems in existing technologies by providing an image classification method based on Bayesian linear regression and a hyperparameter-optimized EM algorithm. This method has rigorous mathematical derivation and explanation, and simulation results demonstrate its fast convergence speed. Using the MNIST classic handwritten digit database, the regression analysis model optimized by the method described in this invention, followed by multi-class logistic regression, is applied to classify and recognize the image test set. The results show more accurate classification and better generalization ability.

[0005] This invention is achieved through the following technical solution: This invention proposes an image classification method based on Bayesian linear regression and hyperparameter optimization of the EM algorithm, the method comprising the following steps:

[0006] Step 1: Obtain the log-marginal likelihood function;

[0007] Step 2: Treat the parameter weights w as latent variables and derive the EM algorithm for hyperparameters α, β, and γ;

[0008] Step 3: Apply the hyperparameter-optimized EM algorithm to image recognition of the MNIST classic handwritten digit database;

[0009] Step 3 specifically involves:

[0010] (1) Each of the 70,000 handwritten digits 0 to 9 has 28×28 pixels. The grayscale data of each image is converted into a one-dimensional vector of 28×28=784 and stored.

[0011] (2) Determine the features of each image from 0 to 9 to obtain the target variable t;

[0012] (3) Apply the EM algorithm to optimize the hyperparameters of the training set;

[0013] (4) For the training set, the least squares method is applied to optimize the weight parameter w and determine the value of the weight parameter w;

[0014] (5) Apply multi-class logistic regression to the optimized hyperparameters and weight parameters w to identify and classify the images of numbers 0 to 9 in 10,000 test sets.

[0015] Further, step 1 specifically includes:

[0016] (1) The prior of the weights is defined as a Gaussian distribution;

[0017] (2) Define a linear regression model;

[0018] (3) Obtain the likelihood function;

[0019] (4) Prove that the posterior distribution of the weights is also a Gaussian distribution;

[0020] (5) Integrate the product of the likelihood function and the Gaussian prior of the weight parameters over the weight parameters to obtain the marginal likelihood function, which is the evidence function, and at the same time obtain the log marginal likelihood function.

[0021] Furthermore, the Gaussian prior of the weights is defined as follows:

[0022]

[0023] Where α=(α1,…,α) M ) T α i It is a hyperparameter and the corresponding parameter w i The precision, γ=(γ1,…,γ M ) T γ i It is a hyperparameter and the corresponding parameter w i The mean;

[0024] The linear regression model is defined as follows:

[0025] A linear regression model is defined as a linear combination of nonlinear functions of the input variables, in the form of:

[0026]

[0027] Where φ i φ(x) is a basis function, w0 is called the bias parameter, and we define φ0(x) = 1 and obtain...

[0028]

[0029] Where w = (w0,…,w M-1 ) T φ(x)=(φ0(x),…,φ M-1 (x)) T .

[0030] Furthermore, the derivation process of the likelihood function is as follows:

[0031] The target variable t is defined by a deterministic function y(x,w) with Gaussian noise.

[0032] t=y(x,w)+ε, (4)

[0033] Where ε is a Gaussian random variable with zero mean and precision β, from which we obtain

[0034] p(t|x,w,β)=N(t|y(x,w),β -1 (5)

[0035] An input dataset X = (x1, ..., x N ) and its corresponding target vector t = (t1, ..., t N The data points are independently sampled from a Gaussian distribution (5), from which the following likelihood function can be obtained.

[0036]

[0037] Furthermore, the proof process is as follows:

[0038] Since (6) is the exponential form of the quadratic function w, the corresponding conjugate prior is defined as

[0039] p(w)=N(w|m0,S0), (7)

[0040] From (6) and (7), the posterior distribution is obtained as follows:

[0041] p(w|t,X,β)=N(w|m N ,S N ), (8)

[0042] here

[0043]

[0044] Where Φ is an N×M matrix, and its elements are defined as Φ nj =φ j (x n ),satisfy

[0045]

[0046] From (6), we can obtain

[0047]

[0048]

[0049] From (1), we can obtain

[0050]

[0051] The posterior distribution of the weights is also a Gaussian distribution, with the following form:

[0052] p(w|t,X,α,β,γ)=N(w|m,Σ), (14)

[0053] Here the mean and covariance are...

[0054] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (15)

[0055] Σ=(A+βΦ T Φ) -1 (16)

[0056] Where A = diag(α) i ).

[0057] Furthermore, the process of integrating the product of the likelihood function and the Gaussian prior of the weight parameters over the weight parameters to obtain the marginal likelihood function, which is also the evidence function, and the specific derivation of the logarithmic marginal likelihood function is as follows:

[0058] First, integrate the weight parameters to obtain the marginal likelihood function, which is also the evidence function.

[0059] p(t|X,α,β,γ)=∫p(t|X,w,β)p(w|α,γ)dw, (17)

[0060] Substituting (12) and (13) into (17), we get

[0061]

[0062] Here is defined

[0063]

[0064] The gradient ∇E(w) of E(w) is:

[0065] ▽E(w)=(A+βΦ T Φ)w-(Aγ+βΦ T t), (20)

[0066] ▽▽E(w)=A+βΦ T Φ=Σ -1 (twenty one)

[0067] let

[0068] ▽E(w)=(βΦ T Φ+A)w-(A+βΦ T t)=0, (22)

[0069] Using both (15) and (22), the stationary point of E(w) can be obtained as follows:

[0070] w=(A+βΦ T Φ) -1 (Aγ+βΦ T t)=m (23)

[0071] Using (21) and (23), we can obtain

[0072]

[0073] here

[0074]

[0075] Substituting (24) into (18) gives

[0076]

[0077]

[0078] here

[0079] From (26), the log-marginal likelihood function is:

[0080]

[0081] Furthermore, step 2 specifically includes:

[0082] ① The joint distribution p(t,w|X,α,β,γ) on t and w is equal to the product of the likelihood function and the prior probability of the weight w, i.e.

[0083] p(t,w|X,α,β,γ)=p(t|X,w,β)p(w|α,γ);

[0084] ② Treating the parameter weights w as latent variables, the log-marginal likelihood function is obtained by integrating the joint distribution p(t,w|X,α,β,γ) on t and w with respect to w.

[0085] L=ln p(t|X,α,β,γ)=ln∫p(t,w|X,α,β,γ)dw;

[0086] ③ Maximize the log-likelihood function;

[0087] ④ Obtain the E-step and M-step of the EM algorithm and the simulation results.

[0088] Furthermore, the derivation process of step ③ is as follows:

[0089] Define a variational probability distribution q(w) with respect to the weights, and use Jensen's inequality to obtain a lower bound for L.

[0090]

[0091] Here, θ represents the remaining parameters. The EM algorithm maximizes the logarithmic marginal likelihood L by iteratively maximizing the lower bound. In the expectation step E, the hyperparameters α, β, and γ are fixed, and F is maximized with respect to the variational distribution q(w). In the maximization step M, with the variational distribution q(w) fixed, F is maximized with respect to the hyperparameters α, β, and γ. The E-step is performed by rewriting the lower bound as F to obtain the desired result.

[0092] F=L+KL(q(w)||p(w|t,X,α,β,γ)), (27)

[0093] Here, KL(q(w)||p(w|t,X,α,β,γ)) is the Kullback-Leibler (KL) divergence between the variational distribution q(w) and the posterior distribution of the weights. The E-step corresponds to the posterior distribution that makes q(w) equal to w, that is, F=L and q(w)=p(w|t,X,α,β,γ). Since the posterior distribution is a Gaussian distribution, the E-step is simplified to calculating its mean m and covariance Σ, which is defined by (15) and (16).

[0094] To achieve the M-step, F is rewritten as...

[0095] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w)), (28)

[0096] Here, H(q(w)) is the information entropy of q(w), and H(q(w)) is a quantity independent of α, β, and γ. Therefore, for the M-step of the EM algorithm, it is necessary to maximize the expected value of the logarithmic joint distribution of t and w in q(w) with respect to α, β, and γ.

[0097] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w))

[0098] =∫q(w)(ln p(w|α,γ)+ln p(t|X,w,β))dw-H(q(w))

[0099] =∫q(w)ln p(w|α,γ)dw+∫q(w)ln p(t|X,w,β)dw-H(q(w)) (29)

[0100] Calculate the first term of (29), and we can obtain

[0101]

[0102] Here, ∫q(w)dw=1

[0103] The third integral of (30) yields...

[0104] ∫q(w)(w-γ) T A(w-γ)dw

[0105] =∫q(w)Tr(A(w-γ)(w-γ) T )dw

[0106] =∫q(w)Tr(Aww) T -wγ T -γw T +γγ T )dw

[0107] =∫q(w)(Tr(Aww)T )-2Tr(wγ T )+Tr(γγ T ))dw

[0108] =∫q(w)Tr(Aww) T )dw-2∫q(w)Tr(wγ T )dw+∫q(w)Tr(γγ T )dw

[0109] =Tr(A∫q(w)ww T dw)-2Tr((∫q(w)wdw)γ T )+Tr(∫q(w)γγ T dw)

[0110] =Tr(AE q (ww T ))-2Tr(E q (w)γ T )+Tr(γγ T )

[0111] =Tr(A(mm) T +Σ))-2Tr(mγ T )+Tr(γγ T )

[0112] =Tr(Amm) T )+Tr(AΣ)-2Tr(mγ T )+Tr(γγ T )

[0113] =m T Am+Tr(AΣ)-2γ T m+γ T γ, (31) Here E q (·) is the mathematical expectation of the random variable on the distribution q(w) and E q (ww T ) = mm T +Σ, Tr(·) is the trace of the matrix, and the mean m and covariance Σ are defined by (15) and (16).

[0114] From (30) and (31), we can obtain

[0115]

[0116] Calculate the second term of (29)

[0117]

[0118] The third integral of (33) yields...

[0119] ∫q(w)(t) T t-2w T F T t+w T F T Φw)dw

[0120] =∫q(w)t T tdw-2∫q(w)w T F T tdw+∫q(w)w T F T Φwdw

[0121] =t T t-2∫q(w)w T F T tdw+vq(w)Tr(Φww T F T )dw

[0122] =t T t-2∫q(w)w T F T tdw+Tr∫q(w)Φww T F T dw

[0123] =t T t-2(∫q(w)w T dw)Φ T t+Tr(Φ(∫q(w)ww T dw)Φ T )

[0124] =t T t-2E q (w T )F T t+Tr(ΦE q (ww T )F T )

[0125] =t T t-2m T F T t+Tr(Φ(mm T +S)F T )

[0126] =t T t-2m T F T t+Tr(Φmm T F T +FSF T )

[0127] =tT t-2m T Φ T t+Tr(Φmm T Φ T )+Tr(ΦΣΦ T )

[0128] =t T t-2m T Φ T t+m T Φ T Φm+Tr(ΦΣΦ T )

[0129] =(t-Φm) T (t-Φm)+Tr(ΦΣΦ T )

[0130] =||t-Φm|| 2 +Tr(ΦΣΦ T )

[0131] =||t-Φm|| 2 +Tr(ΣΦ T Φ)

[0132] =||t-Φm 2 +Tr(Φ T ΦΣ), (34)

[0133] Here E q (ww T ) = mm T +Σ.

[0134] Using (33) and (34), we can obtain

[0135]

[0136] From (32) and (35), we can obtain

[0137]

[0138] Thus, we obtain (36) for α i The stationary points of β and γ satisfy

[0139]

[0140] Therefore, the hyperparameter update equation is obtained as follows:

[0141]

[0142] γ new =m, (42)

[0143] Here mi It is the i-th element of the posterior distribution mean m defined by (15), Σ ii It is the i-th element on the main diagonal of the posterior covariance Σ defined by (16).

[0144] Furthermore, step ④ specifically includes:

[0145] (1) Randomly generate the values ​​of hyperparameters α, β, γ and the dataset X = {x1, ..., x...} N};

[0146] (2) By using (11), select a suitable linear basis function φ(x) n To construct an N×M matrix Φ;

[0147] (3) Through (1), the random sampling weight parameter w;

[0148] (4) Generate an n-dimensional vector ε randomly sampled from a Gaussian distribution N(0,β), and then generate t = (t1,…,t) using formulas (3) and (4). N );

[0149] (5) Step E: Using the values ​​of the current hyperparameters generated in step (1), estimate the mean m and covariance Σ of the posterior distribution defined in (14).

[0150] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (43)

[0151] Σ=(A+βΦ T Φ) -1 (44)

[0152] (6) M-step: Using the mean m and covariance Σ obtained in step (5) and the following update equation, re-estimate the hyperparameters.

[0153]

[0154] γ new =m (47)

[0155] (7) Estimating the log-likelihood

[0156]

[0157] Then check the convergence of the hyperparameters or log-likelihood. If the convergence criterion is not met, return to step (5).

[0158] The beneficial effects of this invention are:

[0159] (1) This invention uses Bayesian theory to derive the EM algorithm, which has a rigorous mathematical interpretation;

[0160] (2) Through simulation results, the EM algorithm has the advantage of fast convergence speed and avoids the disadvantages of brute-force search of some classic algorithms;

[0161] (3) For the MNIST classic handwritten digit database, the regression analysis model optimized by the method described in this invention is then used to classify and recognize the image test set by multi-class logistic regression. From the results, the classification is more accurate and has better generalization ability. Attached Figure Description

[0162] Figure 1 The convergence graph of the EM algorithm;

[0163] Figure 2 Here is the flowchart for the EM algorithm;

[0164] Figure 3 This is a flowchart of an image classification algorithm. Detailed Implementation

[0165] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0166] See Figures 1-3 This invention proposes an image classification method based on Bayesian linear regression and hyperparameter optimization of the EM algorithm. The method includes the following steps:

[0167] Step 1: Obtain the log-marginal likelihood function;

[0168] Step 1 specifically involves:

[0169] (1) The prior of the weights is defined as a Gaussian distribution;

[0170] (2) Define a linear regression model;

[0171] (3) Obtain the likelihood function;

[0172] (4) Prove that the posterior distribution of the weights is also a Gaussian distribution;

[0173] (5) Integrate the product of the likelihood function and the Gaussian prior of the weight parameters over the weight parameters to obtain the marginal likelihood function, which is the evidence function, and at the same time obtain the log marginal likelihood function.

[0174] The Gaussian prior definition of the weights is:

[0175]

[0176] Where α=(α1,…,α) M ) T α i It is a hyperparameter and the corresponding parameter w i The precision, γ=(γ1,…,γ M ) T γ i It is a hyperparameter and the corresponding parameter w i The mean;

[0177] The linear regression model is defined as follows:

[0178] A linear regression model is defined as a linear combination of nonlinear functions of the input variables, in the form of:

[0179]

[0180] Where φ i φ(x) is a basis function, w0 is called the bias parameter, and we define φ0(x) = 1 and obtain...

[0181]

[0182] Where w = (w0,…,w M-1 ) T φ(x)=(φ0(x),…,φ M-1 (x)) T .

[0183] The derivation process of the likelihood function is as follows:

[0184] The target variable t is defined by a deterministic function y(x,w) with Gaussian noise.

[0185] t=y(x,w)+ε, (51)

[0186] Where ε is a Gaussian random variable with zero mean and precision (the reciprocal of variance) β, from which we obtain

[0187] p(t|x,w,β)=N(t|y(x,w),β -1 (52)

[0188] An input dataset X = (x1, ..., x N ) and its corresponding target vector t = (t1, ..., t N The data points are independently sampled from a Gaussian distribution (5), from which the following likelihood function can be obtained.

[0189]

[0190] The proof process is as follows:

[0191] Since (6) is the exponential form of the quadratic function w, the corresponding conjugate prior is defined as

[0192] p(w)=N(w|m0,S0), (54)

[0193] From (6) and (7), the posterior distribution is obtained as follows:

[0194] p(w|t,X,β)=N(w|m N ,S N ), (55)

[0195] here

[0196]

[0197] Where Φ is an N×M matrix, and its elements are defined as Φ nj =φ j (x n ),satisfy

[0198]

[0199] From (6), we can obtain

[0200]

[0201] From (1), we can obtain

[0202]

[0203]

[0204] The posterior distribution of the weights is also a Gaussian distribution, with the following form:

[0205] p(w|t,X,α,β,γ)=N(w|m,Σ), (61)

[0206] Here the mean and covariance are...

[0207] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (62)

[0208] Σ=(A+βΦ T Φ) -1 (63)

[0209] Where A = diag(α) i ).

[0210] The process of integrating the product of the likelihood function and the Gaussian prior of the weight parameters over the weight parameters to obtain the marginal likelihood function, which is also the evidence function, and the specific derivation of the logarithmic marginal likelihood function is as follows:

[0211] First, integrate the weight parameters to obtain the marginal likelihood function, which is also the evidence function.

[0212] p(t|X,α,β,γ)=∫p(t|X,w,β)p(w|α,γ)dw, (64)

[0213] Substituting (12) and (13) into (17), we get

[0214]

[0215] Here is defined

[0216]

[0217] The gradient ∇E(w) of E(w) is:

[0218] ▽E(w)=(A+βΦ T Φ)w-(Aγ+βΦ T t), (67)

[0219] ▽▽E(w)=A+βΦ T Φ=Σ -1 (68)

[0220] let

[0221] ▽E(w)=(βΦ T Φ+A)w-(A+βΦ T t)=0, (69)

[0222] Using both (15) and (22), the stationary point of E(w) can be obtained as follows:

[0223] w=(A+βΦ T Φ) -1 (Aγ+βΦ T t)=m (70)

[0224] Using (21) and (23), we can obtain

[0225]

[0226] here

[0227]

[0228] Substituting (24) into (18) gives

[0229]

[0230] here

[0231] From (26), the log-marginal likelihood function is:

[0232]

[0233] Step 2: Treat the parameter weights w as latent variables and derive the EM algorithm for hyperparameters α, β, and γ;

[0234] Step 2 specifically involves:

[0235] ① The joint distribution p(t,w|X,α,β,γ) on t and w is equal to the product of the likelihood function and the prior probability of the weight w, i.e.

[0236] p(t,w|X,α,β,γ)=p(t|X,w,β)p(w|α,γ);

[0237] ② Treating the parameter weights w as latent variables, the log-marginal likelihood function is obtained by integrating the joint distribution p(t,w|X,α,β,γ) on t and w with respect to w.

[0238] L=ln p(t|X,α,β,γ)=ln∫p(t,w|X,α,β,γ)dw;

[0239] ③ Maximize the log-likelihood function;

[0240] ④ Obtain the E-step and M-step of the EM algorithm and the simulation results.

[0241] The derivation process of step ③ is as follows:

[0242] Define a variational probability distribution q(w) with respect to the weights, and use Jensen's inequality to obtain a lower bound for L.

[0243]

[0244] Here, θ represents the remaining parameters. The EM algorithm maximizes the logarithmic marginal likelihood L by iteratively maximizing the lower bound. In the expectation step E, the hyperparameters α, β, and γ are fixed, and F is maximized with respect to the variational distribution q(w). In the maximization step M, with the variational distribution q(w) fixed, F is maximized with respect to the hyperparameters α, β, and γ. The E-step is performed by rewriting the lower bound as F to obtain the desired result.

[0245] F=L+KL(q(w)||p(w|t,X,α,β,γ)), (74)

[0246] Here, KL(q(w)||p(w|t,X,α,β,γ)) is the Kullback-Leibler (KL) divergence between the variational distribution q(w) and the posterior distribution of the weights. The KL divergence is always a positive number, and it is equal to zero only when the two distributions are exactly the same. The E-step corresponds to the posterior distribution that makes q(w) equal to w, that is, F=L and q(w)=p(w|t,X,α,β,γ). Since the posterior distribution is a Gaussian distribution, the E-step is simplified to calculating its mean m and covariance Σ. This covariance is defined by (15) and (16).

[0247] To achieve the M-step, F is rewritten as...

[0248] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w)), (75)

[0249] Here, H(q(w)) is the information entropy of q(w), and H(q(w)) is a quantity independent of α, β, and γ. Therefore, for the M-step of the EM algorithm, it is necessary to maximize the expected value of the logarithmic joint distribution of t and w in q(w) with respect to α, β, and γ.

[0250] F=∫q(w)ln p(t,w|X,α,β,γ)dw-H(q(w))

[0251] =∫q(w)(ln p(w|α,γ)+ln p(t|X,w,β))dw-H(q(w))

[0252] =∫q(w)ln p(w|α,γ)dw+∫q(w)ln p(t|X,w,β)dw-H(q(w)) (76)

[0253] Calculate the first term of (29), and we can obtain

[0254]

[0255] Here, ∫q(w)dw=1

[0256] The third integral of (30) yields...

[0257] ∫q(w)(w-γ) T A(w-γ)dw

[0258] =∫q(w)Tr(A(w-γ)(w-γ) T )dw

[0259] =∫q(w)Tr(Aww) T -wγ T -γw T +γγ T )dw

[0260] =∫q(w)(Tr(Aww) T )-2Tr(wγ T )+Tr(γγ T ))dw

[0261] =∫q(w)Tr(Aww) T )dw-2∫q(w)Tr(wγ T )dw+∫q(w)Tr(γγ T )dw

[0262] =Tr(A∫q(w)ww T dw)-2Tr((∫q(w)wdw)γ T )+Tr(∫q(w)γγ T dw)

[0263] =Tr(AE q (ww T ))-2Tr(E q (w)γ T )+Tr(γγ T )

[0264] =Tr(A(mm) T +Σ))-2Tr(mγ T )+Tr(γγ T )

[0265] =Tr(Amm) T )+Tr(AΣ)-2Tr(mγ T )+Tr(γγ T )

[0266] =m T Am+Tr(AΣ)-2γ T m+γ T γ, (78) Here E q (·) is the mathematical expectation of the random variable on the distribution q(w) and E q (ww T ) = mm T +Σ, Tr(·) is the trace of the matrix, and the mean m and covariance Σ are defined by (15) and (16).

[0267] From (30) and (31), we can obtain

[0268]

[0269] Calculate the second term of (29)

[0270]

[0271] (33).

[0272] ∫q(w)(t) T t-2w T F T t+w T F T Φw)dw

[0273] =∫q(w)t T tdw-2∫q(w)w T F T tdw+∫q(w)w T F T Φwdw

[0274] =t T t-2∫q(w)w T F T tdw+∫q(w)Tr(Φww T F T )dw

[0275] =t T t-2∫q(w)w T F T tdw+Tr∫q(w)Φww T F T dw

[0276] =t T t-2(∫q(w)w T dw)Φ T t+Tr(Φ(∫q(w)ww T dw)Φ T )

[0277] =t T t-2E q (w T )F T t+Tr(ΦE q (ww T )F T )

[0278] =t T t-2m T F T t+Tr(Φ(mm T +S)F T )

[0279] =t T t-2m T F T t+Tr(Φmm T F T +FSFT )

[0280] =t T t-2m T Φ T t+Tr(Φmm T Φ T )+Tr(ΦΣΦ T )

[0281] =t T t-2m T Φ T t+m T Φ T Φm+Tr(ΦΣΦ T )

[0282] =(t-Φm) T (t-Φm)+Tr(ΦΣΦ T )

[0283] =||t-Φm|| 2 +Tr(ΦΣΦ T )

[0284] =||t-Φm|| 2 +Tr(ΣΦ T Φ)

[0285] =||t-Φm|| 2 +Tr(Φ T ΦΣ), (81)

[0286] Here E q (ww T ) = mm T +Σ.

[0287] Using (33) and (34), we can obtain

[0288]

[0289] From (32) and (35), we can obtain

[0290]

[0291] Thus, we obtain (36) for α i The stationary points of β and γ satisfy

[0292]

[0293] Therefore, the hyperparameter update equation is obtained as follows:

[0294]

[0295] γnew =m, (89)

[0296] Here m i It is the i-th element of the posterior distribution mean m defined by (15), Σ ii It is the i-th element on the main diagonal of the posterior covariance Σ defined by (16).

[0297] Step ④ specifically involves:

[0298] (1) Randomly generate the values ​​of hyperparameters α, β, γ and the dataset X = {x1, ..., x...} N};

[0299] (2) By using (11), select a suitable linear basis function φ(x) n To construct an N×M matrix Φ;

[0300] (3) Through (1), the random sampling weight parameter w;

[0301] (4) Generate an n-dimensional vector ε randomly sampled from a Gaussian distribution N(0,β), and then generate t = (t1,…,t) using formulas (3) and (4). N );

[0302] (5) Step E: Using the values ​​of the current hyperparameters generated in step (1), estimate the mean m and covariance Σ of the posterior distribution defined in (14).

[0303] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (90)

[0304] Σ=(A+βΦ T Φ) -1 (91)

[0305] (6) M-step: Using the mean m and covariance Σ obtained in step (5) and the following update equation, re-estimate the hyperparameters.

[0306]

[0307]

[0308] γ new =m (94)

[0309] (7) Estimating the log-likelihood

[0310]

[0311] Then check the convergence of the hyperparameters or log-likelihood. If the convergence criterion is not met, return to step (5).

[0312] This invention applies the obtained EM algorithm to optimize the values ​​of hyperparameters α, β, and γ. Here, the linear basis function φ(x) is a Gaussian function, making it difficult to determine the convergence of hyperparameters α, β, and γ. Therefore, the convergence of log-likelihood is assessed instead, which is reasonable because the convergence of hyperparameters α, β, and γ implies that the log-likelihood remains unchanged; in other words, the convergence of log-likelihood is much easier to implement in programming. Figure 1 From this, we can conclude that the EM algorithm obtained through rigorous mathematical and statistical interpretation has the advantage of fast convergence.

[0313] Step 3: Apply the hyperparameter-optimized EM algorithm to image recognition of the MNIST classic handwritten digit database;

[0314] Step 3 specifically involves:

[0315] (1) Each of the 70,000 handwritten digits 0 to 9 (including 60,000 training images and 10,000 test images) contains 28×28 pixels. The grayscale data of each image is converted into a one-dimensional vector of 28×28=784 and stored.

[0316] (2) Determine the features of each image from 0 to 9 to obtain the target variable t;

[0317] (3) Apply the EM algorithm to optimize the hyperparameters of the training set;

[0318] The specific steps of step (3) are as follows:

[0319] 1) Construct a dataset X = {x1,…,x...} from 60,000 training images. 60000} and the corresponding target set t=(t1,…,t N );

[0320] 2) Select a suitable linear basis function φ(x) using formula (11). n To construct matrix Φ,

[0321] 3) Using formula (1), randomly sample the weight parameter w.

[0322] 4) Step E: Using the values ​​of the current hyperparameters generated in step 1), estimate the mean and covariance of the posterior distribution defined by formula (14). Estimate the mean m and covariance Σ.

[0323] m=(A+βΦ T Φ) -1 (Aγ+βΦ T t), (95)

[0324] Σ=(A+βΦ T Φ) -1 (96)

[0325] 5) M-step: Using the mean m and covariance Σ obtained in step 4), re-estimate the hyperparameters using the update equation below.

[0326]

[0327] γ new =m (99)

[0328] 6) Estimating the log-likelihood

[0329]

[0330] Then check the convergence of the hyperparameters or log-likelihood. If the convergence criterion is not met, return to step 4.

[0331] (4) For the training set, the least squares method is applied to optimize the weight parameter w and determine the value of the weight parameter w;

[0332] (5) Apply multi-class logistic regression to the optimized hyperparameters and weight parameters w to identify and classify the images of numbers 0 to 9 in 10,000 test sets.

[0333] The above description merely illustrates preferred embodiments of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An image classification method based on Bayesian linear regression and hyperparameter optimization EM algorithm, characterized in that: The method includes the following steps: Step 1: Obtain the log-marginal likelihood function; Step 2: Calculate the parameter weights Treat them as latent variables and derive hyperparameters The EM algorithm; Step 3: Apply the hyperparameter-optimized EM algorithm to image recognition of the MNIST classic handwritten digit database; Step 1 specifically involves: (1) The prior of the weights is defined as a Gaussian distribution; (2) Define the linear regression model; (3) Obtain the likelihood function; (4) Prove that the posterior distribution of the weights is also a Gaussian distribution; (5) Integrate the product of the likelihood function and the Gaussian prior of the weight parameters over the weight parameters to obtain the marginal likelihood function, which is the evidence function, and at the same time obtain the log marginal likelihood function. The Gaussian prior definition of the weights is: ,(1) in , It is a hyperparameter and a corresponding parameter accuracy, , It is a hyperparameter and a corresponding parameter The mean; The linear regression model is defined as follows: A linear regression model is defined as a linear combination of nonlinear functions of the input variables, in the form of: ,(2) in These are basis functions. Called the deviation parameter, defined And obtain ,(3) in , ; The derivation process of the likelihood function is as follows: Target variable From a deterministic function with Gaussian noise Defined ,(4) in It has a mean of zero and a precision of Gaussian random variables, from which we obtain (5) An input dataset and its corresponding target vector The data points are independently sampled from a Gaussian distribution (5), from which the following likelihood function can be obtained. ，(6) The proof that the posterior distribution of the proof weights is also a Gaussian distribution is as follows: Because (6) is a quadratic function The exponential form of is given, so the corresponding conjugate prior is defined as . ，(7) From (6) and (7), the posterior distribution is obtained as follows: ,(8) here ,(9) ,(10) in yes A matrix, whose elements are defined as , satisfy (11) From (6), we can obtain (12) From (1), we can obtain (13) The posterior distribution of the weights is also a Gaussian distribution, with the following form: ,(14) Here the mean and covariance are... ,(15) ,(16) in ; The process of integrating the product of the likelihood function and the Gaussian prior of the weight parameters over the weight parameters to obtain the marginal likelihood function, which is also the evidence function, and the specific derivation of the logarithmic marginal likelihood function is as follows: First, integrate the weight parameters to obtain the marginal likelihood function, which is also the evidence function. ，(17) Substituting (12) and (13) into (17), we get ,(18) Here is defined ，(19) gradient for ,(20) (21) let ,(22) By using (15) and (22) simultaneously, we can obtain The outpost is (23) Using (21) and (23), we can obtain ，(24) here ，(25) Substituting (24) into (18) gives ,(26) here ， From (26), the log-marginal likelihood function is: ； Step 2 specifically involves: ① and Joint distribution on Equals the likelihood function and weight The product of prior probabilities, i.e. ； ② Weight the parameters Treat it as a latent variable, for and Joint distribution on about Integrating, we obtain the log-marginal likelihood function. ； ③ Maximize the log-likelihood function; ④ Obtain the E-step and M-step of the EM algorithm and the simulation results; Step 3 specifically involves: (1) Each of the 70,000 handwritten digits 0 to 9 is Each pixel converts the grayscale data of an image into a single pixel. Store it as a one-dimensional vector; (2) Determine the features of each image from 0 to 9 to obtain the target variable. ; (3) Apply the EM algorithm to optimize the hyperparameters of the training set; (4) Apply the least squares method to the weight parameters on the training set. Optimize and determine weight parameters The value; (5) Optimized hyperparameters and weight parameters The regression analysis model was used, and then multi-class logistic regression was applied to identify and classify the digits 0 to 9 in 10,000 test set images.

2. The method according to claim 1, characterized in that, The derivation process of step ③ is as follows: Define a variational probability distribution with respect to the weights. And using Jensen's inequality, we obtain a lower bound , here The remaining parameters represent the log-marginal likelihood, and the EM algorithm achieves this by iteratively maximizing the lower bound. Maximizing the hyperparameters in the expected step E, with the hyperparameters fixed. , and Regarding variational distribution maximize In the maximization step M, the variational distribution is fixed. In the case of hyperparameters , and To maximize, perform an E-step by rewriting the lower bound as To obtain ,(27) here In variational distribution The Kullback-Leibler (KL) divergence between the posterior distributions of the weights and the weights, the E-step corresponding to making equal The posterior distribution, that is and Since the posterior distribution is Gaussian, the E-step simplifies to calculating its mean. With covariance This covariance is defined by (15) and (16); To achieve M-step, rewrite for ,(28) here yes Information entropy Is with , and Irrelevant quantities; therefore, for the M-step of the EM algorithm, it is necessary to... , and Maximize about and exist The expected value of the log-joint distribution, (29) Calculate the first term of (29) ,(30) here , The third integral of (30) yields... ,(31) here It is a random variable in the distribution Mathematical expectation and , Let be the trace of the matrix, and also the mean. With covariance Defined by (15) and (16); From (30) and (31), we can obtain (32) Calculate the second term of (29) (33) The third integral of (33) yields... ,(34) here ; Using (33) and (34), we can obtain (35) From (32) and (35), we can obtain (36) Therefore, we obtain (36) for , and The station meets ,(37) ,(38) ．(39) Therefore, the hyperparameter update equation is obtained as follows: ,(40) ,(41) ，(42) here It is the mean of the posterior distribution defined by (15). The One element, It is the posterior covariance defined by (16) The first on the main diagonal Each element.

3. The method according to claim 2, characterized in that, Step ④ specifically involves: (1) Randomly generate hyperparameters , , Values ​​and datasets ; (2) Select appropriate linear basis functions using formula (11). To construct matrix ; (3) Using formula (1), the random sampling weight parameters ; (4) Generate a Gaussian distribution Random sampling dimensional vector Then, generate using formulas (3) and (4). ; (5) Step E: Using the values ​​of the current hyperparameters generated in step (1), estimate the mean and covariance of the posterior distribution defined in (14). With covariance ,(43) (44) (6) M step: using the mean obtained in step (5) With covariance Re-estimate hyperparameters using the update equation below ,(45) ,(46) (47) (7) Estimating the log-likelihood Then check the convergence of the hyperparameters or log-likelihood. If the convergence criterion is not met, return to step (5).

Images