Image recognition method based on multi-fidelity bayesian optimization

Abstract

The application discloses an image recognition method based on multi-fidelity Bayesian optimization. Before data fitting, a sick condition number detection is performed, so that the algorithm inference effect is improved, the robustness of the algorithm calculation process is increased, and the final model effect is improved. The intermediate information in the neural network, i.e. error value and main eigenvalue of Hessian matrix, is combined as fidelity, so that the effectiveness of multi-fidelity is improved, the model fitting effect is enhanced, the time consumption of the Bayesian optimization algorithm is reduced, and the generalization of the model prediction is finally improved. The time consumption problem of the Bayesian optimization is solved, so that the time consumption acceptable migration learning ability is realized, and the data privacy is ensured and the labor cost is saved through automation training.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image processing, and particularly relates to an image recognition method based on multi-fidelity Bayesian optimization. BACKGROUND

[0002] The technical problems of the current image recognition technology based on a convolutional neural network include the following aspects: 1. The optimal hyperparameters are different according to data changes, and it is difficult to specify them in advance, which limits the model to achieve the best effect; 2. In the image service, data is often updated, and the corresponding recognition model needs to be retrained, and the time-consuming of the head training is long, which cannot meet the business needs. 3. Image data is often private, which limits the frequency of expert intervention in model debugging, and an automatic method is urgently needed.

[0003] In the model training process, the hyperparameters play a major guiding role. The current automatic method is mainly based on the Bayesian method, but the Bayesian method has the problem of time consumption of waiting for complete model training, which cannot meet the actual needs. The multi-fidelity technology improves the time consumption of the Bayesian method and accelerates the search process, but the traditional multi-fidelity variable cannot meet the actual needs. The application combines the multi-fidelity technology and the intermediate output of the neural network, and proposes a new hyperparameter optimization method to solve the above problems. SUMMARY

[0004] The application proposes an image recognition method based on multi-fidelity Bayesian optimization to solve the time consumption problem of the Bayesian optimization, so as to realize the time consumption acceptable transfer learning ability, and through the training automation, the data privacy is protected and the labor cost is saved.

[0005] The application is implemented by the following technical solutions:

[0006] The application relates to an image recognition method based on multi-fidelity Bayesian optimization, which comprises the following steps:

[0007] Step 1) According to the hyperparameter search range, randomly sample from the hyperparameter search domain, train M neural networks in parallel each time, train one step (period) for each network, respectively obtain an observation result (prediction error), and then calculate the initial observation set by using the current Hessian matrix main eigenvalue calculated by the trained neural network.

[0008] The neural network adopts a ResNet-50 image classification network and an EAST text recognition network, which are respectively recorded in He K, Zhang X, Ren S, et al. Deep Residual Learning for Image Recognition [C]. in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016: 770-778. and Zhou X, Yao C, Wen H, et al. EAST: An Efficient and Accurate Scene Text Detector [C]. in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): vol. 2017. 2017: 2642-2651.

[0009] The main eigenvalue of the Hessian matrix is calculated by the following method:

[0010] i) twice back propagation derivation is performed on the neural network to obtain the Hessian matrix of the network;

[0011] ii) a vector with the same rank as the Hessian matrix is randomly initialized;

[0012] iii) matrix multiplication is performed on the vector and the Hessian matrix to obtain a new vector;

[0013] iv) it is judged whether the difference between the vectors before and after multiplication is less than a threshold value;

[0014] v) steps iii) to iv) are repeated until a preset number of times or the difference is less than a preset threshold value, and a final principal eigenvector is obtained;

[0015] vi) the main eigenvalue is calculated according to the Hessian matrix and the principal eigenvector.

[0016] Step 2) The observation set is tested to determine whether there is a problem of ill-conditioned condition number; when the condition number exceeds a threshold value, the observation set is screened to reduce the condition number of the covariance matrix.

[0017] The ill-conditioned condition number problem refers to that, for a Gaussian process-based Bayesian optimization algorithm, the distance between observation points is too dense or unevenly distributed, the covariance matrix of the Gaussian process will rapidly degenerate into a non-positive definite matrix, thereby causing the square root decomposition method to fail.

[0018] The covariance matrix condition number is reduced, that is, the threshold is set as the logarithm of the condition number being less than or equal to 20; when the logarithm is greater than 20, all observation points are clustered, and part of the points that are too close are deleted, and the remaining points are disturbed, so that the logarithm of the condition number is effectively reduced from 20-25 to 15-18.

[0019] Step 3) In each inference period, a Gaussian process is used to fit the observation set based on a product kernel composed of two variables of hyperparameters and neural network states, to obtain a variable posterior distribution, and then a next query point is inferred using an expected acquisition function.

[0020] The product kernel is: κ((x,s),(x',s'))=κ1(x,x')κ2(s,s'), wherein: both Gaussian kernels are Marton kernels, that is, θ0 and λ are preset hyperparameters, d λ (x,x')=(x-x') T diag(λ)(x-x') represents the Mahalanobis distance.

[0021] The observation fitting refers to: according to the observation data set and the selected Marton kernel, a covariance matrix is calculated, and the posterior distribution of the query point can be calculated from the covariance matrix in subsequent calculation, so as to perform Bayesian inference.

[0022] Preferably, based on the assumption that the optimal hyperparameters are always continuously changed during the training process, the query range of the acquisition function in the inference process is limited to the range of 20% of the search region length centered on the optimal point of the current optimal hyperparameter.

[0023] Step 4) In each query period, M neural networks trained in parallel in step 1 are used for training to obtain a prediction error and a current Hessian matrix principal eigenvalue.

[0024] The training refers to: under the condition that the experimental conditions are kept the same, the neural network is trained according to the hyperparameters inferred by the Bayesian algorithm, including the learning rate, the training batch, and the weight decay.

[0025] Step 5) The latest observation point is added to the observation set, and steps 2 to 4 are repeated until the M neural networks are fully trained, and the searched hyperparameter group is image-identified.

[0026] Technical effects

[0027] The present application solves the problem that the fidelity is not effective enough in the prior art, the modeling effect on the neural network training process is poor, thereby affecting the inference performance, and finally reducing the accuracy of image recognition.

[0028] Compared with the prior art, the present application detects ill-conditioned condition number before data fitting, thereby improving algorithm inference effectiveness, increasing algorithm calculation process robustness, and improving final model effect. The intermediate information in the neural network, i.e. error value and Hessian matrix main eigenvalue, is combined as fidelity, thereby improving multi-fidelity effectiveness, enhancing model fitting effect, reducing Bayesian optimization algorithm time consumption, and finally improving model prediction generalization. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0030] As Figure 1 shown, the present embodiment relates to an image recognition method based on multi-fidelity Bayesian optimization, including the following steps:

[0031] Step 1) According to the hyperparameter search range, randomly sample from the hyperparameter search domain, respectively use the neural network to train one step, obtain the prediction error, then calculate the Hessian matrix main eigenvalue at this time, thereby obtaining the initial observation set, the specific operation steps include:

[0032] 1.1 Initialize the neural network, obtain the training error l0 on the training set, calculate the Hessian matrix main eigenvalue λ0 on the validation set, but do not perform back propagation.

[0033] 1.2 Randomly sample M*N groups of initial observation hyperparameters from the search domain. M represents the number of parallel training models, N represents the training period, M and N are specified by experts in advance, and usually M*N >= (hyperparameter dimension + fidelity dimension)*3.

[0034] 1.3 In each training period, the neural network is trained M times in parallel, and each time only one group of hyperparameters x mn is taken. Thus, the neural network loss function value l mn is obtained. The cross-entropy loss function is used for classification tasks, and the mean square error loss function is used for regression tasks. Test on the validation set to obtain the corresponding performance y mn of the model, and perform two derivations on the neural network to obtain the Hessian matrix, and use the power iteration algorithm to calculate the main eigenvalue λ mn .

[0035] 1.4 Record all observation values, which are composed of {(x mn , l mn-1 , λ mn-1 ), y mn}, select the best-performing model (the one with the highest accuracy or recall rate on the validation set), and replace the rest with its weight.

[0036] Step 2) Examine the observation set to determine if there is an ill-conditioned condition number problem; if the condition number exceeds a threshold, filter the observation set to reduce the condition number of the covariance matrix. Specific steps include:

[0037] 2.1 Calculate the covariance matrix corresponding to the observation set using the Gaussian process kernel function, and then calculate the L-2 conditional logarithm of this matrix.

[0038] 2.2 Determine if the conditional logarithm exceeds a threshold (this threshold is specified by experts and is usually set to 20). If it does, cluster the observation points. If there are more observation points in one category (more than twice the number of points in other categories), downsampling is performed, half of the points are deleted, and the remaining points are randomly perturbed.

[0039] 2.3 Repeat steps 2.1-2.2 until the conditional logarithm is below the threshold.

[0040] Step 3) In each inference cycle, the observation set is fitted using a Gaussian process based on the product kernel of two variables: hyperparameters and neural network state, to obtain the posterior distribution of the variables. Then, the next query point is inferred using an expectation-based acquisition function. The specific steps include:

[0041] 3.1 Calculate the covariance matrix K(X, X) = κ(X, X) corresponding to the observation points {X, y};

[0042] 3.2 Randomly sample L points (L is usually 500 * hyperparameter dimension) on the search domain to obtain the query set X. * And calculate the covariance matrix K(X, X). * )=K(X * ,X)=κ(X,X * ) and K(X * X * )=κ(X * X * ), where κ() is the kernel function; thus, the query set X is calculated. * The corresponding predicted posterior distribution is That is, the predicted values ​​are distributed by a multivariate Gaussian distribution, where: the mean function * indicates a prediction. The superscript indicates the mean, and the covariance function is cov(f * )=K(X * X * )-K(X * ,X)[K(X,X)+σ 2 I] -1 K(X, X) * Let X represent the input set of observed data, y be its corresponding output (accuracy in classification tasks, recall in recognition tasks), and σ be the input set of observed data.2 σ2is the variance of the noise Gaussian distribution, usually set to 1 (Note: the mean of the noise Gaussian distribution is usually set to 0), -1 is to get the inverse of the matrix.

[0043] 3.3 According to the predicted posterior distribution, use the acquisition function Wherein: represents the current best prediction performance, here * represents the best, t-1 represents the best value in the data point from the observed point rather than the current best, Φ, φ respectively represent the cumulative distribution function and the probability density function, And cov(f * ) has been described in

[0040] , γ(x) as a whole represents the improvement of the current prediction optimal value compared with the observed optimal value. Solve the value of x that makes EI(x) maximum as the recommended value of the next training period.

[0044] Step 4) In each query period, use the neural network to train to get the prediction error and the current Hessian matrix principal eigenvalue, the specific operation steps include:

[0045] 4.1 According to the hyperparameter setting x, usually including learning rate, training period, momentum, weight decay, etc., train a cycle of neural network, so as to obtain the prediction error and the current Hessian matrix principal eigenvalue.

[0046] 4.2 Add new observation points to the observation set, repeat steps 3.1-3.3.

[0047] Step 5) Add the latest observation points to the observation set, and repeat steps 2 to 4 until the model is fully trained, the specific operation steps include:

[0048] 5.1 Repeat steps 2-4 T times, T is specified by experts. Model training stops when the number of training reaches T, or the accuracy / recall rate changes less than the threshold (0.01 or 0.001) on the validation set for five times in a row.

[0049] 5.2 Use the fully trained model to perform image recognition tasks, and its performance is improved.

[0050] Through specific experiments, the server used has 128G memory, the CPU is Intel(R) Core(TM) i9-7920X 2.GHz, the GPU is NVIDIA GeForce RTX 2080 Ti 11 GB, the algorithm uses python3.8.3 and pytorch experiments. Training ResNet-50 model on CIFAR-10 dataset for 160 cycles, the method (M = 3, T = 160) can obtain 92.89% accuracy (28.8 hours), expert debugging (random search algorithm, random 50 times) is only 91.12% (170 hours), and FABOLAS algorithm (Klein A, Falkner S, Bartels S, et al. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets [C]. in: International Conference on Artificial Intelligence and Statistics (AISTATS 2017). 2016: 528-536.) is only 88.98% (27.6 hours); training ResNet-50 model on CIFAR-100 dataset for 200 cycles, the method (M = 3, T = 200) can obtain 74.67% accuracy (36.7 hours), expert debugging (random search algorithm, random 50 times) is only 71.43% (251 hours), and FABOLAS algorithm is only 72.92% (35.3 hours); training EAST model on ICDAR-2015 dataset, the method (M = 3, T = 100) can obtain 81.27% recall rate (35.6 hours), expert debugging (random search algorithm, random 50 times) is only 72.75% (240 hours), and FABOLAS algorithm is only 78.65% (34.01 hours).

[0051] Compared with the traditional Bayesian method, the time efficiency of the application is greatly improved, which is about five times the effect; compared with the classic multi-fidelity Bayesian algorithm FABOLAS, the accuracy of the image recognition model is improved with less time difference.

[0052] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the application, the protection scope of the application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraints of the application.

Claims

1. An image recognition method based on multi-fidelity Bayesian optimization, characterized in that, include: Step 1) Based on the hyperparameter search range, randomly sample from the hyperparameter search domain, train M neural networks in parallel each time, train each network for one step size, obtain a prediction error, and then calculate the main eigenvalue of the current Hessian matrix through the trained neural network to obtain the initial observation set; Step 2) Examine the initial observation set from Step 1 to determine if there is an ill-conditioned condition number problem; When the condition number exceeds the threshold, the observation set is filtered to reduce the condition number of the covariance matrix; Step 3) In each inference cycle, the observation set in Step 2 is fitted using a Gaussian process based on the product kernel composed of two variables: hyperparameters and neural network state, to obtain the posterior distribution of the variables. Then, the next query point is inferred using the acquisition function based on expectation. Step 4) In each query cycle, train the M neural networks trained in parallel in Step 1 to obtain the prediction error and the current principal eigenvalues ​​of the Hessian matrix, where: M represents the number of models trained in parallel and satisfies M*N>=(hyperparameter dimension + fidelity dimension)*3, and N represents the training cycle. Step 5) Add the latest observation point to the observation set in Step 3, and repeat Steps 2 to 4 until the M neural networks are fully trained, and then perform image recognition on the searched hyperparameter set.

2. The image recognition method based on multi-fidelity Bayesian optimization according to claim 1, characterized in that, The neural network described uses the ResNet-50 image classification network and the EAST text recognition network.

3. The image recognition method based on multi-fidelity Bayesian optimization according to claim 1, characterized in that, The principal eigenvalues ​​of the Hessian matrix are calculated in the following manner: ① By performing two backpropagation operations on the neural network, the derivative is obtained to obtain the Hessian matrix of the network; ② Randomly initialize vectors of the same rank as the Hessian matrix; ③ Perform matrix multiplication between the vector and the Hessian matrix to obtain a new vector; ④ Determine whether the difference between the vectors before and after multiplication is less than a threshold; ⑤ Repeat steps ③-④ until the preset number of times is completed or the difference is less than the preset threshold, to obtain the final main feature vector; ⑥ Calculate the principal eigenvalues ​​based on the Hessian matrix and principal eigenvectors.

4. The image recognition method based on multi-fidelity Bayesian optimization according to claim 1, characterized in that, The reduction of the condition number of the covariance matrix refers to: setting a threshold where the logarithm of the condition number is less than or equal to 20; when the logarithm is greater than 20, clustering all observation points and deleting some overly close points, while perturbing the remaining points, thereby effectively reducing the logarithm of the condition number from 20-25 to 15-18.

5. The image recognition method based on multi-fidelity Bayesian optimization according to claim 1, characterized in that, The product kernel is: Where both Gaussian kernels are Marton kernels, i.e. , and To preset hyperparameters, This represents the Mahalanobis distance.

6. The image recognition method based on multi-fidelity Bayesian optimization according to claim 1, characterized in that, The aforementioned fitted observation refers to: calculating a covariance matrix based on the observation dataset and the selected Marton kernel, and then using this covariance matrix to calculate the posterior distribution of the query point in subsequent calculations, thereby performing Bayesian inference.

7. The image recognition method based on multi-fidelity Bayesian optimization according to claim 1, characterized in that, The training mentioned refers to training the neural network based on hyperparameters inferred by the Bayesian algorithm, including learning rate, training batch, and weight decay, while keeping the experimental conditions the same.

8. The image recognition method based on multi-fidelity Bayesian optimization according to any one of claims 1 to 7, characterized in that, Step 1 specifically includes: 1.1 Initialize the neural network and obtain the training error on the training set. The principal eigenvalues ​​of the Hessian matrix were calculated on the validation set. However, it does not perform backpropagation; 1.2 Based on random sampling of the search domain, M*N initial observation hyperparameter sets are obtained; 1.3 In each training cycle, the neural network is trained in parallel M times, with only one set of hyperparameters used each time. Thus, the numerical value of the neural network loss function is obtained. The classification task uses the cross-entropy loss function, while the regression task uses the mean squared error loss function. The model is then tested on a validation set to obtain its performance. The neural network is differentiated twice to obtain the Hessian matrix, and the principal eigenvalues ​​are calculated using a power iteration algorithm. ; 1.4 Record all observations, which consist of {( ), The best-performing neural network, i.e. the one with the highest accuracy or recall on the validation set, is selected and its weights are replaced with those of the others.

9. The image recognition method based on multi-fidelity Bayesian optimization according to any one of claims 1 to 7, characterized in that, Step 2 is described in detail include: 2.1 Calculate the covariance matrix corresponding to the observation set using the Gaussian process kernel function, and then calculate the L-2 conditional logarithm of this matrix; 2.2 Determine if the number of conditional logarithms exceeds 20; if it does, cluster the observation points. If the number of observation points in one class is more than twice the number of points in other classes, downsample the data, delete half of the points, and randomly perturb the remaining points. 2.3 Repeat steps 2.1-2.2 until the conditional logarithm is below the threshold.

10. The image recognition method based on multi-fidelity Bayesian optimization according to any one of claims 1 to 7, characterized in that, Step 2 specifically includes: 3.1 Calculate the covariance matrix corresponding to the observation points {X,y} ; 3.2 Randomly sample L points on the search domain to obtain the query set. And calculate the covariance matrix. as well as Where: L = 500 * hyperparameter dimension, The kernel function is used to compute the query set. The corresponding predicted posterior distribution is That is, the multivariate Gaussian distribution of the predicted values, where: mean function * indicates a prediction. The superscript indicates the mean and covariance function. X represents the input set of observed data, and y is its corresponding output, which is accuracy in classification tasks and recall in recognition tasks. This represents the variance of the Gaussian distribution of the noise. 3.3 Based on the predicted posterior distribution, a data acquisition function is adopted. ,in: This represents the best predictive performance to date. These are the cumulative distribution function and the probability density function, respectively. Find the x value that maximizes EI(x) and use it as the recommended value for the next training cycle.

Images