Distributed model training algorithm for fast image classification problem
By introducing Nesterov momentum and gradient tracking mechanisms in the Adam algorithm, the distributed optimization algorithm NGTAdam is improved, and the problems of slow update speed and poor convergence of image classification tasks in the existing technology are solved, achieving more efficient image classification processing and better classification accuracy.
Patent Information
- Application Number
- CN202411116682.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-15
AI Technical Summary
The existing distributed optimization algorithms are slow to update and poor convergence in dynamic environments, especially in online learning scenarios, making it difficult to effectively handle large-scale image classification tasks.
A new distributed online optimization algorithm NGTAdam is proposed. By introducing Nesterov momentum variables and gradient tracking mechanisms, the Adam algorithm is improved, the adaptability to dynamic changes of network structures is increased, and the processing speed and classification accuracy of image classification tasks are improved.
The NGTAdam algorithm significantly improves computing efficiency and classification accuracy when processing large-scale image data, has better performance and efficiency, and theoretically guarantees convergence. It is suitable for parallel processing of multiple computing nodes and online optimization scenarios.
Smart Images

Figure CN119006913A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed optimization and machine learning, and in particular to a distributed model training algorithm for fast image classification problems. Background Art
[0002] Image classification is one of the core tasks in the field of computer vision, and image classification technology is a key factor in promoting the development of computer vision and machine learning. Image classification technology is widely used in various industries. In scenarios such as medical diagnosis and emergency response systems that require rapid decision-making, image classification can quickly provide decision support and help professionals make accurate judgments in the shortest possible time; in scientific research fields such as biology, geography, and astronomy, image classification technology can help researchers automatically identify and classify large amounts of image data and promote scientific discovery; with the explosive growth of digital images and video content, image classification technology has become an effective tool for managing large-scale data. With the continuous advancement of technology and algorithms, image classification will play a greater role in more fields.
[0003] At present, there is an urgent need to accelerate the processing speed of image classification tasks and improve classification accuracy, and various image classification methods are constantly developing. Deep neural networks, such as convolutional neural networks, perform well in image recognition and classification tasks. However, training these complex networks requires a lot of computing resources and optimization algorithms. Traditional optimization algorithms, such as gradient descent, are less efficient when processing large-scale and high-dimensional image data. Therefore, it is necessary to seek more efficient optimization methods to accelerate the training process and improve the performance of the model. As an adaptive learning rate optimization algorithm, the Adam algorithm combines the momentum method and adaptive learning rate technology to effectively improve the convergence speed and performance of deep learning models. However, as the data set and model size grow, the traditional Adam algorithm may still have the problem of low optimization efficiency when processing large-scale image classification tasks.
[0004] To solve the above problems, distributed optimization algorithms have emerged. They improve computing efficiency by processing data in parallel on multiple computing nodes. In recent years, distributed optimization technology has been widely used in machine learning, image processing, network optimization, intelligent control and other fields.
[0005] However, existing distributed optimization algorithms often face the problems of slow update speed and poor convergence in dynamic environments, especially in online learning scenarios. In order to improve the performance of distributed optimization algorithms in image classification tasks, there is an urgent need for a new algorithm that can quickly adapt to data changes, effectively utilize local information, and maintain algorithm stability. Summary of the invention
[0006] In view of the deficiencies of the above-mentioned prior art, the present invention provides a distributed model training algorithm for the problem of fast image classification: NGT Adam (Nesterov Accelerated Gradient Tracking with Adaptive Momentum) algorithm, which is a new distributed online optimization algorithm. It accelerates and improves the Adam algorithm commonly used in image classification in the prior art based on the Nesterov momentum variable, increases the adaptability to dynamic changes of the network structure, accelerates the processing speed of image classification tasks, and improves the classification accuracy.
[0007] In order to achieve the above technical objectives, the present invention provides the following technical solutions:
[0008] A distributed model training algorithm for fast image classification problems, including:
[0009] S1. Initialize the parameters of the NGT Adam algorithm; import the image data set and perform data preprocessing, and distribute the training data to all nodes on the decentralized network;
[0010] S2. At each node, the temporary variables are updated using the estimated solution and the Nesterov momentum variable, and the current gradient is calculated using the temporary variables.
[0011] S3. At each node, update the gradient estimator based on the gradient tracking method;
[0012] S4. At each node, the first-order moment estimate and the second-order matrix estimate are updated using exponential moving average;
[0013] S5. At each node, adaptively update the gradient descent direction of the estimated solution using the first-order moment estimate and the second-order moment estimate updated in step S4;
[0014] S6. Combine the Nesterov momentum variables of neighboring nodes and update the Nesterov momentum variables of this node based on the gradient descent method;
[0015] S7, combining the temporary variables of neighboring nodes, and updating the estimated solution of this node based on the gradient descent method;
[0016] S8. Repeat S2-S7 for multiple iterations until the algorithm termination condition is met.
[0017] Preferably, step S2 specifically includes:
[0018] S21, update temporary variables; use the estimated solution and Nesterov momentum variables to update the temporary variables of node i at time t Its formula is:
[0019]
[0020] Among them, θ t represents the decreasing step size, is the estimated solution of node i at time t, is the estimated Nesterov momentum variable of node i at time t;
[0021] S22, calculate the current gradient; use the temporary variable obtained in step S21 Calculate the gradient of node i at time t Its formula is:
[0022]
[0023] in, is the gradient calculation function.
[0024] Preferably, step S3 is specifically:
[0025] The gradient estimator of node i at time t is recorded as It is used to indicate the update descent direction of the Nesterov momentum variable, and its update expression is:
[0026]
[0027] Among them, N i represents the neighborhood of node i, j is the neighborhood of node i in N i Neighbor nodes in ; are the gradient w of node i at time t and time t-1 respectively. ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of node i on the information of its neighbor node j.
[0028] Preferably, step S4 specifically includes:
[0029] S41, update the first-order moment estimate; node i uses exponential moving average to update the first-order moment estimate, and the first-order moment estimate of node i at time t+1 is recorded as Its update expression is:
[0030]
[0031] Where β1 represents the exponential decay rate of the first-order moment estimate, is the first-order moment estimate of node i at time t, is the gradient estimator of node i at time t;
[0032] S42, update the second-order moment estimate; node i uses exponential moving average to update the second-order moment estimate, and the second-order moment estimate of node i at time t+1 is recorded as Its update expression is:
[0033]
[0034] Where β2 represents the exponential decay rate of the second-order moment estimate, the symbol ⊙ represents the Hadamard product of the matrix, and V m >>0 is the saturation parameter, which guarantees the upper bound of the estimated value; is the second-order moment estimate of node i at time t, is the gradient estimator of node i at time t.
[0035] Preferably, step S5 is specifically:
[0036] The descent direction of the estimated solution is adaptively updated using the first-order moment estimate and the second-order moment estimate updated in step S4. The gradient descent direction of the estimated solution of node i at time t+1 is recorded as Its update expression is:
[0037]
[0038] Among them, the parameter ∈>0 is used to ensure the stability of numerical operations and avoid gradient explosion; are the first-order moment estimate and the second-order moment estimate of node i at time t+1, respectively.
[0039] Preferably, step S6 specifically includes:
[0040] Node i combines the Nesterov momentum variables of neighboring nodes and updates its own Nesterov momentum variables based on the gradient descent method. The Nesterov momentum variable of node i at time t+1 is recorded as Its update expression is:
[0041]
[0042] Among them, N i is the neighborhood of node i, j is the neighborhood of node i in domain N i Neighbor nodes within is the Nesterov momentum variable of node i’s neighbor node j at time t; α is a fixed step size, θ t is the decreasing step size, w ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of node i on its neighbor node j; gradient estimator Indicates the updated descent direction.
[0043] Preferably, step S7 specifically includes:
[0044] Node i combines the temporary variables of neighbor node j and updates the estimated solution based on the gradient descent method; the estimated solution of node i at time t+1 is recorded as Its update expression is:
[0045]
[0046] Among them, N i is the neighborhood of node i, j is the neighborhood of node i in domain N i Neighbor nodes within, w ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of the information of node i on its neighbor node j; is the temporary variable of i’s neighbor node j at time t, α is a fixed step size, is the gradient descent direction of the estimated solution of node i at time t+1.
[0047] Preferably, step S8 is specifically:
[0048] Each repetition of steps S2-S7 is recorded as an iteration. After each iteration, the optimal error of the loss function corresponding to the estimated solution of the current output needs to be evaluated. If it does not reach the set error range, the iteration is repeated until the set error range or the set maximum number of iterations is reached, and the algorithm training is terminated.
[0049] More specifically, the decrement step length θ in the above steps S2 and S6 is t Specifically:
[0050]
[0051] Wherein, t represents time t;
[0052] The fixed step size α in step S6 and step S7 selects different specific values for different classification problems, and the specific selection range is: α∈{0.001, 0.01, 0.1}.
[0053] Furthermore, the above-mentioned NGTAdam algorithm is used to realize rapid classification of images, which specifically includes the following steps:
[0054] (1) Establish an image dataset by using the collected real image data to establish the dataset, label the training samples and test samples in the image dataset, and form a training set and a test set;
[0055] (2) Import the data set to be distributed and perform data preprocessing, and distribute the data to all nodes on the decentralized network;
[0056] (3) Initialize the NGT Adam algorithm model parameters at each node;
[0057] (4) Construct a distributed optimization problem based on the NGTAdam algorithm and define the objective function of the optimization problem as the average hinge loss of all data points combined with the regularization term, that is:
[0058]
[0059] in, w represents the weight vector to be optimized and corresponds to the estimated solution finally output by the NGTAdam algorithm; c i Represents data point b i The category or classification target, N represents the total number of nodes in the distributed system, and λ is the regularization parameter; through the function f i (w) Calculate the loss of the i-th node to evaluate the optimization performance.
[0060] (5) Import the image dataset and use the NGT Adam algorithm for training in the framework of distributed optimization problem;
[0061] (6) Output classification results and evaluate the accuracy of image classification;
[0062] (7) Deploy the trained model to the actual environment to perform actual image classification tasks.
[0063] Based on the above technical solution, the present invention has the following beneficial effects:
[0064] The distributed model training algorithm (NGTAdam algorithm) proposed in the present invention improves the Adam algorithm commonly used in image classification, introduces the Nesterov momentum and gradient tracking mechanism, and effectively solves the problems of large computing resources and low optimization efficiency required by existing optimization algorithms when processing large-scale image classification tasks. In addition, the present invention provides theoretical support for the NGT Adam algorithm to solve online optimization problems, so that its convergence can be guaranteed.
[0065] Compared with several existing image classification technologies, the present invention has the following advantages:
[0066] (1) Compared with the centralized algorithm, the NGTAdam algorithm provided by the present invention can process data in parallel on multiple computing nodes, breaking through the computational bottleneck of the centralized algorithm, which makes it more efficient in processing large-scale image data sets.
[0067] (2) Compared with the existing distributed algorithms, the NGTAdam algorithm provided by the present invention combines the advantages of Nesterov momentum, gradient tracking mechanism and adaptive moment estimation method. While ensuring the accuracy of image classification, it greatly improves the speed of image classification and has better performance and efficiency when processing large-scale image data sets.
[0068] (3) The NGTAdam algorithm provided by the present invention is scalable and can be applied not only to image classification tasks, but also directly to all online optimization scenarios where the objective function is convex. In addition, the NGTAdam algorithm can adjust the number of iterations and the convergence threshold according to the complexity of the data model, and adjust the step size according to the specific scenario to ensure optimal performance.
[0069] (4) The convergence of the NGT Adam algorithm provided by the present invention is theoretically guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 It is the flow chart of the NGTAdam algorithm of the present invention;
[0071] Figure 2 Flowchart for performing image classification tasks for the NGT Adam algorithm;
[0072] Figure 3 A comparison chart of the results of NGT Adam algorithm performing multi-type image classification tasks;
[0073] Figure 4 This is a comparison chart of the results of the NGT Adam algorithm performing the image binary classification task. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0075] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.
[0076] The invention proposes a distributed model training algorithm for fast image classification problems, the core of which is the Nesterov accelerated gradient tracking adaptive moment estimation algorithm: NGTAdam. The NGTAdam algorithm is a distributed online optimization algorithm that adds Nesterov momentum and gradient tracking mechanisms to accelerate the Adam algorithm and accelerate the completion of image classification tasks.
[0077] like Figure 1 As shown, a distributed model training algorithm for fast image classification problem proposed by the present invention is given, which specifically includes:
[0078] S1. Initialize the parameters of the NGT Adam algorithm; import the image data set and perform data preprocessing, and distribute the training data to all nodes on the decentralized network;
[0079] In this embodiment, the image classification problem is divided into: multi-type image classification and image binary classification;
[0080] The parameters for initializing the NGTAdam algorithm include: the number of nodes N, the decay rates β1, β2, and the decreasing step length θ t , parameter ∈, first-order moment estimate Second moment estimate Gradient Estimator gradient and estimated solution Different image problems are assigned different initial values:
[0081] For multi-type image classification: set the number of nodes N = 40, decay rate β1 = 0.9, β2 = 0.999, model parameter dimension q = 100, regularization parameter λ = 0.1; after parameter adjustment, set the fixed step size α = 0.01, and the decreasing step size It should be noted that the specific step size is adjusted according to the classification task of the image dataset; setting the first-order moment estimate Second moment estimate Nesterov Momentum Variable Temporary variables All elements of are 0, the gradient estimator The estimated solution Arbitrary setting; parameter ∈ = 10 -4 , convergence threshold tol_optgap = 10 -24 , the mini-batch size is set to 10 and the maximum number of iterations is set to 100.
[0082] For binary image classification: set the number of nodes N = 10, the fixed step size α = 0.1, and the remaining parameters are consistent with the multi-type image classification task.
[0083] In addition, in this embodiment, the seed command of the random number generator: rng(238) is used to ensure that the same random number sequence is obtained each time the program is run, thereby ensuring that repeatable random optimization experiments can be performed and improving the rigor of the experimental results.
[0084] After initializing the parameters, data preprocessing is performed, and different preprocessing methods are used for different problems:
[0085] In the embodiment, the MNIST dataset and the Mushroom dataset are selected to test the image classification performance. These two datasets represent the classic problems of image recognition and classification tasks in machine learning, and correspond to typical applications in the fields of computer vision and data mining, respectively.
[0086] Multi-type image classification: The multi-type image classification task is evaluated on the MNIST dataset. The MNIST dataset contains grayscale images of handwritten digits with a pixel size of 28×28, ranging from 0 to 9. In the training set, there are 6,000 training images for model training. In the test set, there are 10,000 test images for evaluating the performance of the model. Each image has a corresponding label, which represents the handwritten digit in the image. The label is an integer between 0 and 9, corresponding to 10 categories.
[0087] Image binary classification: The image binary classification task is evaluated on the Mushroom dataset, using a linear support vector machine (Linear SVM). The Mushroom dataset is a classic binary classification problem dataset, which contains 8124 samples. Each sample has 22 features describing the properties of the mushroom, such as the shape, color, smell, etc., and a label indicating whether the mushroom is edible. The dataset is divided into a training set and a test set to evaluate the training effect and generalization ability of the model.
[0088] Import the MNIST and Mushroom datasets respectively, and read useful information from the datasets, including the feature matrices of training data and test data: x_train and x_test, the labels of training data and test data: y_train and y_test, the optimal weight vector w_opt, and the number of categories l.
[0089] After the initialization and preprocessing in step S1 above, the model training begins.
[0090] S2. At each node, the temporary variables are updated using the estimated solution and the Nesterov momentum variable, and the current gradient is calculated using the temporary variables.
[0091] More specifically, step S2 includes:
[0092] S21, update temporary variables; use the estimated solution and Nesterov momentum variables to update the temporary variables of node i at time t Its formula is:
[0093]
[0094] Among them, θ t represents the decreasing step size, is the estimated solution of node i at time t, is the estimated Nesterov momentum variable of node i at time t;
[0095] S22, calculate the current gradient; use the temporary variable obtained in step S21 Calculate the gradient of node i at time t Its formula is:
[0096]
[0097] in, is the gradient calculation function.
[0098] S3. On each node, update the gradient estimator based on the gradient tracking method:
[0099] The gradient estimator of node i at time t is recorded as Its update expression is:
[0100]
[0101] Among them, N i represents the neighborhood of node i, j is the neighborhood of node i in N i Neighbor nodes in ; are the gradient w of node i at time t and time t-1 respectively. ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of node i on the information of its neighbor node j.
[0102] Gradient Estimator It is used to indicate the update descent direction of the Nesterov momentum variable, which is conducive to accelerating convergence. It also directly determines the update direction of the first-order moment estimate, the second-order moment estimate and the Nesterov momentum variable, and indirectly affects the update of the learning rate and the estimated solution.
[0103] S4. At each node, the first-order moment estimate and the second-order matrix estimate are updated using exponential moving average;
[0104] More specifically, step S4 includes:
[0105] S41, update the first-order moment estimate; node i uses exponential moving average to update the first-order moment estimate, and the first-order moment estimate of node i at time t+1 is recorded as Its update expression is:
[0106]
[0107] Where β1 represents the exponential decay rate of the first-order moment estimate, is the first-order moment estimate of node i at time t, is the gradient estimator of node i at time t;
[0108] S42, update the second-order moment estimate; node i uses exponential moving average to update the second-order moment estimate, and the second-order moment estimate of node i at time t+1 is recorded as Its update expression is:
[0109]
[0110] Where β2 represents the exponential decay rate of the second-order moment estimate, the symbol ⊙ represents the Hadamard product of the matrix, and V m >>0 is the saturation parameter, which guarantees the upper bound of the estimated value; is the estimated value of the second moment of node i at time t.
[0111] S5. At each node, use the first-order moment estimate and the second-order moment estimate updated in step S4 to adaptively update the gradient descent direction of the estimated solution:
[0112] The descent direction of the estimated solution is adaptively updated using the first-order moment estimate and the second-order moment estimate updated in step S4. The gradient descent direction of the estimated solution of node i at time t+1 is recorded as Its update expression is:
[0113]
[0114] Among them, the parameter ∈>0 is used to ensure the stability of numerical operations and avoid gradient explosion; are the first-order moment estimate and the second-order moment estimate of node i at time t+1, respectively.
[0115] S6. Combine the Nesterov momentum variables of neighboring nodes and update the Nesterov momentum variables of this node based on the gradient descent method:
[0116] Node i combines the Nesterov momentum variables of neighboring nodes and updates its own Nesterov momentum variables based on the gradient descent method. The Nesterov momentum variable of node i at time t+1 is recorded as Its update expression is:
[0117]
[0118] Among them, N i is the neighborhood of node i, j is the neighborhood of node i in domain N i Neighbor nodes within is the Nesterov momentum variable of node i’s neighbor node j at time t; α is a fixed step size, θ t is the decreasing step size, w ijIt is an element in the weight matrix W of the time-varying network, reflecting the weight of node i on its neighbor node j; gradient estimator Indicates the updated descent direction.
[0119] S7. Combine the temporary variables of neighboring nodes and update the estimated solution of this node based on the gradient descent method:
[0120] Node i combines the temporary variables of neighbor node j and updates the estimated solution based on the gradient descent method; the estimated solution of node i at time t+1 is recorded as Its update expression is:
[0121]
[0122] Among them, N i is the neighborhood of node i, j is the neighborhood of node i in domain N i Neighbor nodes within, w ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of the information of node i on its neighbor node j; is the temporary variable of i’s neighbor node j at time t, α is a fixed step size, is the gradient descent direction of the estimated solution of node i at time t+1.
[0123] S8. Repeat S2-S7 for multiple iterations until the algorithm termination condition is met.
[0124] More specifically, in this example, steps S2 to S7 are recorded as an iterative process of the NGT Adam algorithm. All nodes 1 to N need to complete one iteration. After completion, the error value is calculated and compared with the set convergence threshold tol_optgap. The error value is selected as needed, for example, the optimal solution error or the optimal loss function error can be selected; when the error value > tol_optgap and the number of iterations < the maximum number of iterations, the iteration is repeated until the set conditions for the iteration are no longer met.
[0125] In addition, in this embodiment, the specific process of using the above-mentioned NGTAdam algorithm for fast image classification is:
[0126] (1) Establish an image dataset by using the collected real image data to establish the dataset, label the training samples and test samples in the image dataset, and form a training set and a test set;
[0127] (2) Import the data set to be distributed and perform data preprocessing, and distribute the data to all nodes on the decentralized network;
[0128] (3) Initialize the NGT Adam algorithm model parameters at each node;
[0129] (4) Construct a distributed optimization problem based on the NGTAdam algorithm and define the objective function of the optimization problem as the average hinge loss of all data points combined with the regularization term, that is:
[0130]
[0131] in, w represents the weight vector to be optimized and corresponds to the estimated solution finally output by the NGTAdam algorithm; c i Represents data point b i The category or classification target, N represents the total number of nodes in the distributed system, and λ is the regularization parameter; through the function f i (w) Calculate the loss of the i-th node to evaluate the optimization performance.
[0132] (5) Import the image dataset and use the NGT Adam algorithm for training in the framework of distributed optimization problem;
[0133] (6) Output classification results and evaluate the accuracy of image classification;
[0134] (7) Deploy the trained model to the actual environment to perform actual image classification tasks.
[0135] In this embodiment, NGT Adam is compared with DAdam and Acc-DNGD algorithms, focusing on the convergence speed and convergence error of the algorithms; DAdam is a distributed Adam algorithm, which is commonly used in image classification tasks and shows the advantages of the adaptive moment estimation algorithm; the Acc-DNGD algorithm is a Nesterov acceleration algorithm based on the gradient descent method, which shows the acceleration advantages of the Nesterov momentum and gradient tracking mechanism; the NGT Adam algorithm proposed in the present invention combines the Nesterov momentum, gradient tracking and adaptive moment estimation methods, and has the advantages of both.
[0136] This embodiment further evaluates the classification accuracy by calculating the optimal value error, outputting the error curve and the classification result.
[0137] Select the optimal loss function error to evaluate the performance of the algorithm. The calculation expression is as follows:
[0138]
[0139] in, Represents the average value of the estimated solutions of all nodes at time t. It is a centralized solution to the optimization problem at time t.
[0140] Figure 3This is a convergence curve for performing multi-type image classification tasks, where the vertical axis represents the error of the optimal loss function and the horizontal axis represents the total number of gradient calculations. Figure 3 It can be seen that DAdam converges faster than Acc-DNGD, which reflects the advantage of adaptive momentum estimation. The convergence speed of NGTAdam is significantly better than that of DAdam and Acc-DNGD, and it can achieve a smaller convergence error in a shorter time.
[0141] Figure 4 This is a convergence curve for performing the binary image classification task. Similarly, the vertical axis represents the error of the optimal loss function and the horizontal axis represents the total number of gradient calculations. Figure 4 It can be seen that the convergence speeds of Acc-DNGD and DAdam are similar. The convergence speed of NGTAdam is significantly better than the other two algorithms, and it can obtain a smaller optimization error in a shorter time. The classification results show that NGTAdam can achieve 100% accuracy in the image binary classification task and 99.897% accuracy in the multi-type image classification task.
[0142] These two embodiments show that the image classification method proposed in the present invention has a faster classification speed while ensuring accurate classification, and has advantages in both image classification efficiency and accuracy.
[0143] In addition, the present invention provides a dynamic Regret upper bound of NGTAdam for online convex optimization problems, indicating that the algorithm is theoretically feasible when dealing with online convex problems.
[0144] According to theoretical analysis, NGT Adam is a dynamic method for solving convex optimization problems. The upper bound is:
[0145]
[0146] Among them, the matrix It is related to the initial conditions of the linear inequality system, Q T and W T Related to the time-varying nature of the online optimization problem. L is the Lipschitz constant of the optimization objective function, It is the second largest eigenvalue of the communication network weight matrix W, and the constant λ>0. Q T and W T All have explicit upper bounds. In the application of this invention, we do not need to pay attention to the specific upper bound expressions, so the upper bound expressions are omitted here.
[0147] In summary, the method proposed in the present invention focuses on real-time decision-making and dynamic optimization, can efficiently solve large-scale online optimization problems, is applicable to a variety of scenarios, and solves the problems of slow speed and poor accuracy of existing image classification algorithms when training large-scale image data.
[0148] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0149] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A distributed model training algorithm for fast image classification problems, characterized by: The distributed model training algorithm is an adaptive moment estimation algorithm based on Nesterov acceleration, referred to as the NGTAdam algorithm, which specifically includes the following steps: S1. Initialize the parameters of the NGT Adam algorithm; import the image dataset and perform data preprocessing, and distribute the training data to all nodes on the decentralized network; S2. At each node, use the estimated solution and Nesterov momentum variable to update the temporary variable, and use the temporary variable to calculate the current gradient; S3. At each node, update the gradient estimator based on the gradient tracking method; S4. At each node, use exponential moving average to update the first-order moment estimate and the second-order matrix estimate; S5. At each node, adaptively update the gradient descent direction of the estimated solution using the first-order moment estimate and the second-order moment estimate updated in step S4. S6. Combine the Nesterov momentum variables of neighboring nodes and update the Nesterov momentum variable of this node based on the gradient descent method; S7, combining the temporary variables of neighboring nodes and updating the estimated solution of this node based on the gradient descent method; S8. Repeat S2-S7 for multiple iterations until the algorithm termination condition is met.
2. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S2 is specifically as follows: S21, update temporary variables; use the estimated solution and Nesterov momentum variables to update the temporary variables of node i at time t Its formula is: Among them, θ t Indicates the decreasing step size, is the estimated solution of node i at time t, is the estimated Nesterov momentum variable of node i at time t; S22, calculate the current gradient; use the temporary variable obtained in step S21 Calculate the gradient of node i at time t Its formula is: in, is the gradient calculation function.
3. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S3 is specifically as follows: The gradient estimator of node i at time t is recorded as It is used to indicate the updated descending direction of the Nesterov momentum variable, and its updated expression is: Among them, N i represents the neighborhood of node i, j is the neighborhood of node i in N i Neighbor nodes in ; are the gradient w of node i at time t and time t-1 respectively ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of node i on the information of its neighbor node j.
4. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S4 specifically includes: S41, update the first-order moment estimate; node i uses exponential moving average to update the first-order moment estimate, and the first-order moment estimate of node i at time t+1 is recorded as Its update expression is: Where β1 represents the exponential decay rate of the first-order moment estimate, is the first-order moment estimate of node i at time t, is the gradient estimator of node i at time t; S42, update the second-order moment estimate; node i uses exponential moving average to update the second-order moment estimate, and the second-order moment estimate of node i at time t+1 is recorded as Its update expression is: Where β2 represents the exponential decay rate of the second-order moment estimate, the symbol ⊙ represents the Hadamard product of the matrix, V m >>0 is a saturation parameter, which guarantees the upper bound of the estimated value; is the second-order moment estimate of node i at time t, is the gradient estimator of node i at time t.
5. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S5 is specifically as follows: The descent direction of the estimated solution is adaptively updated using the first-order moment estimate and the second-order moment estimate updated in step S4. The gradient descent direction of the estimated solution of node i at time t+1 is recorded as Its update expression is: Among them, the parameter ∈>0 is used to ensure the stability of numerical operations and avoid gradient explosion; are the first-order moment estimate and second-order moment estimate of node i at time t+1, respectively.
6. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S6 is specifically as follows: Node i combines the Nesterov momentum variables of neighboring nodes and updates its own Nesterov momentum variables based on the gradient descent method. The Nesterov momentum variable of node i at time t+1 is recorded as Its update expression is: Among them, N i is the neighborhood of node i, j is the neighborhood of node i in the domain N i Neighbor nodes within is the Nesterov momentum variable of node i’s neighbor node j at time t; α is a fixed step size, θ t is the decreasing step size, w ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of node i on the information of its neighbor node j.
7. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S7 is specifically as follows: Node i combines the temporary variables of neighbor node j and updates the estimated solution based on the gradient descent method; the estimated solution of node i at time t+1 is recorded as Its update expression is: Among them, N i is the neighborhood of node i, j is the neighborhood of node i in the domain N i Neighbor nodes within, w ij It is an element in the weight matrix W of the time-varying network, reflecting the weight of the information of node i on its neighbor node j; is the temporary variable of i’s neighbor node j at time t, α is a fixed step size, is the gradient descent direction of the estimated solution of node i at time t+1.
8. A distributed model training algorithm for fast image classification problem according to claim 2, 6 or 7, characterized in that: The decrement step θ in step S2 and step S6 t Specifically: Wherein, t represents time t; The fixed step size α in step S6 and step S7 selects different specific values for different classification problems. The specific selection range is: α∈{0.001, 0.01, 0.1}.
9. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: Step S8 is specifically as follows: Each repetition of steps S2-S7 is recorded as an iteration. After each iteration, the optimal error of the loss function corresponding to the estimated solution of the current output needs to be evaluated. If it does not reach the set error range, the iteration is repeated until the set error range or the set maximum number of iterations is reached, and the algorithm training ends.
10. A distributed model training algorithm for fast image classification problem according to claim 1, characterized in that: The NGTAdam algorithm is used to achieve rapid image classification, which includes the following steps: (1) Establish an image dataset using the collected real image data, label the training samples and test samples in the image dataset, and form a training set and a test set; (2) Import the data set to be distributed and perform data preprocessing, and distribute the data to all nodes on the decentralized network; (3) Initialize the NGT Adam algorithm model parameters at each node; (4) Construct a distributed optimization problem based on the NGT Adam algorithm and define the objective function of the optimization problem as the average hinge loss of all data points combined with the regularization term, that is: in: w represents the weight vector to be optimized corresponding to the estimated solution finally output by the NGT Adam algorithm; c i Represents data point b i The category or classification target, N represents the total number of nodes in the distributed system, and λ is the regularization parameter; through the function f i (w) Calculate the loss of the i-th node to evaluate the optimization performance; (5) Import the image dataset and use the NGTAdam algorithm for training in the framework of distributed optimization problem; (6) Output classification results and evaluate the accuracy of image classification; (7) Deploy the trained model to the actual environment to perform actual image classification tasks.
Citation Information
Patent Citations
Long-time visual target tracking method based on continuous learning
CN110728694A
Intelligent medical image analysis system based on distributed deep learning
CN112988382A
Distributed artificial intelligence (AI) / machine learning training system
CN114787833A
Adaptive optimization training method of convolutional neural network for image classification
CN115205577A
Proximal Gradient Method for Huberized Support Vector Machine
US20150262083A1
Cited By
DMG-MPC algorithm-based logistics AGV formation cooperative control method
CN120972927A
