An image classification method based on variable step size ADMM algorithm extreme learning machine

Abstract

The application discloses an image classification method based on a variable step ADMM algorithm extreme learning machine, proposes a variable step alternating direction method of multipliers (Var-ADMM), and through dynamic adjustment of an iteration step of a solving process, defects of slow convergence of an ADMM algorithm are made up, and operation precision is effectively improved; through combination of the variable step ADMM algorithm and ELM, a regularized method is used to limit the size of a weight coefficient in the process of solving an output weight matrix, over-sensitivity of a model to input containing noise due to excessively large weight coefficients is avoided, recognition accuracy is reduced, over-fitting of ELM is effectively inhibited, and the classification accuracy of ELM on images is improved.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology in computer vision, and relates to image classification optimization methods, specifically to an image classification method based on the variable step size ADMM algorithm extreme learning machine. Background Technology

[0002] Image recognition and classification are key and hot topics in computer vision research, with wide applications across various industries. For example, in the medical field, they assist in disease diagnosis, including but not limited to tumor detection and lesion analysis; in the transportation field, they are used for license plate recognition and traffic sign recognition; and in the security field, they are used for facial recognition and behavior analysis. In general, the development of image recognition and classification technology has brought tremendous changes and opportunities to many fields. With the continuous advancement of algorithms and models, image recognition and classification will play an even more important role in more areas, bringing greater convenience and innovation to people's lives.

[0003] The core of image recognition and classification technology lies in deep learning and neural networks. By constructing deep neural network models and training them with large amounts of labeled data, computers can extract features from images, learn patterns, and make accurate predictions. Commonly used image recognition and classification algorithms include Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). However, the training process of neural networks uses the backpropagation algorithm, which leverages gradient descent to continuously reduce the error between the model's output and the actual labels through multiple iterations of learning, thus completing the model training. As the number of network layers in a neural network model increases, the parameters that need to be adjusted also grow exponentially. This makes parameter training not only time-consuming and laborious but also poses significant challenges to researchers' model validation and debugging processes. Even minor changes to the model design require extensive training to see the final effect, rather than immediate results. Therefore, some scholars are considering improvements to the model structure and training methods of deep learning to address the current predicament.

[0004] Extreme Learning Machine (ELM) is one such solution. ELM is a single-hidden-layer feedforward neural network algorithm. Due to its simple structure, random generation of weights and biases of hidden layer nodes during training, and determination of output weights by solving the least-norm solution of a system of linear equations, ELM can achieve training speed improvements of hundreds or thousands of times compared to many traditional machine learning algorithms. Moreover, it can achieve good generalization performance while maintaining high learning accuracy, and is increasingly being used in the fields of image recognition and classification.

[0005] However, due to the random generation of hidden layer parameters in ELM, the accuracy of the trained network model is highly unstable. Furthermore, since the model training process is based on the minimum empirical risk criterion—that is, the ELM model aims to minimize training error—the model is prone to overfitting when the training error is small. This is because the model overlearns the features of the training set and ignores the differences between training and test samples, resulting in suboptimal accuracy on the test set, decreased generalization performance, and an inability to accurately classify and recognize images. Therefore, it is necessary to consider using regularization methods to suppress overfitting.

[0006] One of the methods for solving the regularization problem of Extreme Learning Machine is the Alternating Direction Method of Multipliers (ADMM). ADMM is a distributed framework for solving linear equality optimization problems. It can decompose the optimization problem with equality constraints into multiple subproblems and solve them alternately. However, this method has the problems of slow iterative convergence speed and stagnation of error decay in the later stage of iteration. Summary of the Invention

[0007] In view of this, the present invention provides an image classification method based on the variable step size ADMM algorithm extreme learning machine, which achieves higher efficiency and higher accuracy in image classification based on existing extreme learning machine-based image classification methods.

[0008] This invention provides an image classification method based on the variable step size ADMM algorithm extreme learning machine, comprising the following steps:

[0009] Step 1: Collect image data to build an image dataset, and label the image data in the image dataset to form a training sample set;

[0010] Step 2: Construct an Extreme Learning Machine (ELM) model based on the Variable Step Size (ADMM) algorithm. The input of the ELM model is the image to be classified, and the output is the classification of the image. The ELM model is shown below:

[0011]

[0012] Among them, β = [β1, β2,···,β L ] T The output weight matrix is ​​T = [t1, t2, ..., t N ] T Let H be the input matrix and H be the hidden layer output matrix. Let f(x) denote the square of the F-norm of the matrix, ||·||1 be the 1-norm of the matrix, and λ>0 and μ>0 be the L1 and L2 regularization coefficients, respectively.

[0013] Step 3: Set the hyperparameters of the Extreme Learning Machine model, namely: the number of hidden layer nodes L, the L1 regularization coefficient λ, the L2 regularization coefficient μ, the initial iteration step size ρ0, and the maximum number of iterations k. max Randomly generate the input weight coefficient matrix w and bias coefficient b of the hidden layer neurons, and randomly generate the initial values ​​y of the three iteration process variables. 0 z 0 u 0 Calculate the hidden layer output matrix H = g(wx + b), where w is the input weight matrix of the hidden layer neurons and the input layer neurons, b is the hidden layer bias coefficient vector, and g(·) is the activation function.

[0014] Step 4: Input the sample images from the training sample set into the initial extreme learning machine model, and replace the weight coefficient matrix β with the introduced iterative process variables y and z to obtain the objective function:

[0015]

[0016] sty-z=0

[0017] Calculate y k+1 =(H T H+(μ+ρ)I) -1 (H T T+ρ(z k -u k ), where I is the identity matrix, z k+1 =S λ / ρ (y k+1 +u k ) and u k+1 =u k +y k+1 -z k+1 We obtain the output weight matrix β; we change the current iteration step size ρ and increment the iteration count by 1; then we calculate the current iteration error. The iteration error is determined when the iteration error meets the condition or the iteration count is greater than k. max If the training of the Extreme Learning Machine model is completed in time, proceed to step 5; otherwise, proceed to step 4.

[0018] Step 5: Input the image to be classified into the trained Extreme Learning Machine model. The Extreme Learning Machine model outputs the category of the image to be classified, thus completing the image classification.

[0019] Furthermore, the method for changing the current iteration step size ρ in step 4 is as follows: Where α is the step size decay rate.

[0020] Furthermore, g(·) is the sigmoid function.

[0021] Beneficial effects:

[0022] This invention improves upon the standard ADMM algorithm by proposing the Variable Step Alternating Direction Method of Multipliers (Var-ADMM). By dynamically adjusting the iteration step size in the solution process, it effectively improves computational accuracy while overcoming the slow convergence of the ADMM algorithm. By combining the Variable Step ADMM algorithm with ELM, regularization is used to limit the size of the weight coefficients during the solution of the output weight matrix. This prevents the model from becoming overly sensitive to noisy inputs due to excessively large weight coefficients, thus avoiding a decrease in recognition accuracy. Consequently, it effectively suppresses overfitting in ELM and improves the classification accuracy of ELM for images. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the network structure of the Extreme Learning Machine.

[0024] Figure 2 The diagram illustrates the principle of image classification using the variable step size ADMM algorithm for extreme learning machines, as provided in this invention. Detailed Implementation

[0025] The following examples illustrate the invention in detail.

[0026] This invention provides an image classification method based on the variable step size ADMM algorithm for extreme learning machines. The basic idea is to introduce L1 and L2 regularization terms on the basis of traditional extreme learning machines, and use the variable step size ADMM algorithm to calculate the output weight matrix to construct an extreme learning machine image classification model based on the variable step size ADMM algorithm. Then, the model is trained using a large number of image samples containing labeled data. After that, the trained model can be used to predict and infer the labels of image samples with unknown labels to obtain the label output, thereby realizing image classification.

[0027] This invention provides an image classification method based on the variable step size ADMM algorithm extreme learning machine, which specifically includes the following steps:

[0028] Step 1: Collect a certain amount of image data and perform category labeling to obtain a dataset. Divide the dataset into a training set and a test set according to a certain ratio, and standardize the feature vectors of the images.

[0029] Specifically, let the input data sample be (x,t), where x is the feature vector of the input data, x = [x1, x2, ..., xt]. n ]T ∈R, n is the feature dimension; t is the label vector of the input data, t=[t1,t2,···,t m ] T ∈R, where m is the label dimension.

[0030] Step 2: Construct an Extreme Learning Machine (ELM) model based on the Variable Step Size ADMM algorithm, as shown in formula (4). Set the six hyperparameters of the ELM model based on the Variable Step Size ADMM algorithm as follows: number of hidden layer nodes L, L1 regularization coefficient λ, L2 regularization coefficient μ, initial iteration step size ρ0, step size decay rate α, and maximum number of iterations k. max .

[0031] The single-hidden-layer neural network structure of the Extreme Learning Machine, such as Figure 1 As shown, assuming the number of neurons in the hidden layer network of the Extreme Learning Machine is L, the mathematical model of the Extreme Learning Machine can be expressed as:

[0032]

[0033] Among them, w i =[w i1 ,w i2 ,···,w in ] T β is the input weight vector connecting the i-th hidden layer neuron and the input layer neuron. i =[β i1 ,β i2 ,···,β im [b] is the output weight vector connecting the i-th hidden layer neuron and the output layer neuron. i Let g be the bias coefficient of the i-th hidden layer neuron, and g(·) be the activation function. The input weight matrix and bias coefficients are randomly generated.

[0034] For a dataset {(x) containing N training samples i ,t i ),i=1,2,···N},x i =[x i1 ,x i2 ,···,x in ] T ∈R,t i =[t i1 ,t i2 ,···,t im ] T Let R ∈, and arrange the outputs of the N data points obtained through the hidden layer neurons into the following matrix form:

[0035]

[0036] Then formula (1) can be rewritten as formula (2):

[0037] Hβ=T (2)

[0038] Among them, β = [β1, β2,···,β L ] T The output weight matrix is ​​T = [t1, t2, ..., t...]. N ] T Let H be the input matrix; H be the hidden layer output matrix. Therefore, the training process of the extreme learning machine is to solve the following unconstrained optimization problem:

[0039]

[0040] in, Let F be the norm of the matrix. Traditional extreme learning machines solve for the least squares solution of equation (3) using the generalized inverse. However, as mentioned earlier, the model parameters obtained using this method have a certain risk of overfitting. Therefore, this invention introduces an L1 penalty term λ||β||1 and an L2 penalty term. By constraining the model parameters, the original optimization problem (3) is transformed into the following optimization problem:

[0041]

[0042] Formula (4) is the optimization problem corresponding to the mathematical model of the Extreme Learning Machine based on the Variable Step Size ADMM algorithm provided by this invention, where ||·||1 represents the 1 norm of the matrix, and λ>0 and μ>0 are the L1 and L2 regularization coefficients, respectively. When the value of λ is small, the L1 regularization effect is not obvious, and the ability to constrain the model complexity is limited. It can increase the expressive power of the model but will bring the risk of overfitting. When the value of λ increases, the L1 regularization effect is enhanced, the ability to constrain the model complexity increases, and the model complexity can be effectively controlled. However, there is also the risk of losing system performance due to the model complexity being too low. Therefore, the optimal value of λ needs to be considered in light of the actual problem and adjusted multiple times to achieve the best balance between model performance and model complexity.

[0043] Step 3: Prepare to start model training. Randomly generate the input weight coefficient matrix w and bias coefficient b of the hidden layer neurons. Select a suitable activation function g(·), usually the sigmoid function, and calculate the hidden layer output matrix H = g(wx + b).

[0044] Step 4: Use the training set generated in Step 1 to train the Extreme Learning Machine model based on the Variable Step Size ADMM algorithm to obtain the output weight matrix β, thereby completing the training of the Extreme Learning Machine model based on the Variable Step Size ADMM algorithm.

[0045] The solution process for the weight matrix β in this invention is as follows: Since the introduction of the L1 regularization term causes the objective function to no longer satisfy the condition of continuous differentiability, this invention uses the ADMM algorithm for solution. First, variables y and z are introduced, and the objective function expression (4) is split into two parts:

[0046]

[0047] Comparing formulas (4) and (5), we can see that the positions of the weight coefficient matrix β are replaced by y and z, respectively. That is, the original problem about β is split into two subproblems about y and z. Thus, the original problem of minimizing the value is transformed into two subproblems. The problem of finding the minimum value of λ||z||1 is solved by keeping the optimal solutions of the two subproblems consistent through constraints. As the iterations alternate, the original problem is solved in a distributed manner.

[0048] Introducing an intermediate variable v for the intermediate transformation of formula (10), the augmented Lagrangian function of problem (5) is as follows:

[0049]

[0050] Where ρ is the iteration step size, for the convenience of subsequent solutions, the scaled form of the augmented Lagrangian function is used, and u = v / ρ is defined. Substituting into equation (6) yields:

[0051]

[0052] Where const is a constant that is independent of parameter updates. According to the ADMM algorithm theory, the optimization problem (7) can be solved by the following iterative formula:

[0053]

[0054]

[0055] u k+1 =u k +y k+1 -z k+1 (10)

[0056] Wherein, the superscript indicates the current iteration number, and the solution to formula (8) can be obtained directly by differentiation:

[0057]

[0058] Solving formula (9) requires the use of the soft threshold function S. λ / ρ (·), the results are as follows:

[0059]

[0060] set up z k+1 y k+1 u k The i-th component, so the specific form of formula (12) is as follows:

[0061]

[0062] Where ρ is the iteration step size. Based on the above analysis, the solution process for the weight coefficient matrix β is summarized as follows: First, set the current iteration number k = 0, and randomly initialize the iteration process variable y. 0 z 0 u 0 The superscript indicates the iteration number; entering the loop begins the iteration:

[0063] Step 4.1, Calculate y k+1 =(H T H+(μ+ρ)I) -1 (H T T+ρ(z k -u k )), z k+1 =S λ / ρ (y k+1 +u k ) and calculation u k+1 =u k +y k+1 -z k+1 .

[0064] Step 4.2: Update the iteration step size ρ using formula (14).

[0065] The iteration step size ρ decreases continuously with the number of iterations. In the early stages of iteration, a larger iteration step size is chosen to reduce the error quickly. However, as the iteration progresses, the degree to which the constraints are satisfied increases, and the error gradually decreases. To avoid oscillations in the convergence process caused by an excessively large step size, the iteration step size is reduced, thus making the convergence process of the ADMM algorithm more stable and faster. The change process is shown below:

[0066]

[0067] Where k is the number of iterations and α is the decay rate, which characterizes how quickly the iteration step size changes with the number of iterations.

[0068] Finally, the optimal solutions of the subproblems solved in each step are made to approach each other using the auxiliary variable iteration formula (10). That is, y and z will continue to approach each other as the iteration progresses until they are equal when they converge. Take β = y = z to obtain the optimal solution β of the original problem, which means the training of the model is completed.

[0069] Step 4.3: Calculate the current iteration error e = ||y k+1 -y k || F ;

[0070] Step 4.4: Determine if the current iteration error is small enough. If the condition is met, exit the loop; otherwise, proceed to step 4.1.

[0071] Figure 2 The image classification principle diagram of this invention is as follows: First, the RGB channels of the input image are separated to obtain a three-dimensional spatial array. Then, the three-dimensional spatial array is concatenated by rows to obtain a two-dimensional array. Finally, the two-dimensional array is expanded by rows to obtain a one-dimensional sequence, which is used as a data input sample and input into the model for calculation. Finally, the category is judged according to the category score, and the category with the highest score is taken as the image category output.

[0072] Step 5: After performing the same standardization process as in Step 1 on the image to be classified, input it into the Extreme Learning Machine model based on the Variable Step Size ADMM algorithm trained in Step 4 for computational inference to obtain the category output of the image to be classified, thus completing the image classification.

[0073] To verify the effectiveness of the image classification method based on the variable step size ADMM algorithm of this invention, experiments were conducted on multiple UCI (University of California, Irvine) datasets (standard datasets commonly used in machine learning) and the MedMNIST medical image dataset released by Shanghai Jiao Tong University. The comparison algorithms were traditional extreme learning machine, regularized extreme learning machine, ADMM-based extreme learning machine, and support vector machine. Compared with existing technologies, the advantages and positive effects of this invention are as follows:

[0074] Compared with traditional extreme learning machines, regularized extreme learning machines, extreme learning machines based on ADMM algorithm, and support vector machines, it has higher classification accuracy and better generalization performance, and can effectively suppress model overfitting.

[0075] It exhibits excellent model training speed on high-dimensional, multi-sample datasets, achieving a speed improvement of up to 76 times compared to Support Vector Machines. This includes performance optimization techniques such as caching used in Support Vector Machines. Compared to Extreme Learning Machines using the standard ADMM algorithm, it can achieve a speed advantage of 3 to 5 times. Moreover, this advantage will become more pronounced as the dataset and sample dimensions increase.

[0076] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image classification method based on the Variable Step Size ADMM algorithm for extreme learning machines, characterized in that, Includes the following steps: Step 1: Collect image data to build an image dataset, and label the image data in the image dataset to form a training sample set; Step 2: Construct an Extreme Learning Machine (ELM) model based on the Variable Step Size (ADMM) algorithm. The input of the ELM model is the image to be classified, and the output is the classification of the image. The ELM model is shown below: Among them, β = [β1, β2,···,β L ] T The output weight matrix is ​​T = [t1, t2, ..., t N ] T Let H be the input matrix and H be the hidden layer output matrix. Let ||·||1| represent the square of the F-norm of the matrix, ||·||1| is the 1-norm of the matrix; λ is the L1 regularization coefficient, and λ > 0; μ is the L2 regularization coefficient, and μ > 0. Step 3: Set the hyperparameters of the Extreme Learning Machine model, namely: the number of hidden layer nodes L, the L1 regularization coefficient λ, the L2 regularization coefficient μ, the initial iteration step size ρ0, and the maximum number of iterations k. max Randomly generate the input weight coefficient matrix w and bias coefficient b of the hidden layer neurons, and randomly generate the initial values ​​y of the three iteration process variables. 0 z 0 u 0 Calculate the hidden layer output matrix H = g(wx + b), where w is the input weight matrix of the hidden layer neurons and the input layer neurons, b is the hidden layer bias coefficient vector, and g(·) is the activation function. Step 4: Input the sample images from the training sample set into the initial extreme learning machine model, and replace the weight coefficient matrix β with the introduced iterative process variables y and z to obtain the objective function: sty-z=0 Calculate y k+1 =(H T H+(μ+ρ)I) -1 (H T T+ρ(z k -u k ), where I is the identity matrix, z k+1 =S λ / ρ (y k+1 +u k ) and u k+1 =u k +y k+1 -z k+1 We obtain the output weight matrix β; we change the current iteration step size ρ and increment the iteration count by 1; then we calculate the current iteration error. The iteration error is considered complete when it is less than a threshold or the iteration count is greater than k. max If the training of the Extreme Learning Machine model is completed in time, proceed to step 5; otherwise, proceed to step 4. Step 5: Input the image to be classified into the trained Extreme Learning Machine model. The Extreme Learning Machine model outputs the category of the image to be classified, thus completing the image classification.

2. The image classification method according to claim 1, characterized in that, The method for changing the current iteration step size ρ in step 4 is as follows: Where α is the step size decay rate.

3. The image classification method according to claim 1, characterized in that, The g(·) is the sigmoid function.

Images