A control method, apparatus, device, and medium for training an image classification model.

Abstract

This application provides a control method, apparatus, device, and medium for training an image classification model, relating to the field of model optimization. The method involves obtaining the objective function, current model parameters, and training dataset of the model to be trained; determining the gradient components of the current model parameters based on the objective function; independently determining the corresponding target first-order momentum based on the gradient components of each current model parameter; scaling the target first-order momentum to obtain the target parameter update amount of the current model parameters; determining the target model parameters based on the target parameter update amount and the current model parameters; and iteratively training the model to be trained using the training dataset under the target model parameters until the target model is obtained. The training process only requires storing the target first-order momentum and model parameters, saving storage resources and reducing training costs. The update of each model parameter depends only on its own gradient component, achieving independent updates of each model parameter, ensuring sufficient model parameter updates, and improving model optimization performance.

Description

Technical Field

[0001] This application relates to the field of model optimization, and in particular to a control method, apparatus, device and medium for training an image classification model. Background Technology

[0002] With the rapid development of large language models, the number of model parameters has increased from millions to hundreds of billions or even trillions, placing higher demands on the computational and storage resources required for training these models. Therefore, optimizing and improving model algorithms has become a key factor in enhancing model training efficiency.

[0003] Currently, Adam (Adaptive Moment Estimation) and AdamW (Adam with Decoupled Weight Decay) adaptive optimizers are commonly used for model training to achieve good convergence. However, training models using these Adam or AdamW optimizers requires storing the first-order momentum (m) and second-order momentum (v). This means that in addition to storing the model's own parameters, at least twice the storage resources are needed to store the optimizer parameters, resulting in extremely high training costs.

[0004] Furthermore, existing techniques utilize fixed-time stable gradient flow methods to determine the update values ​​of model parameters, ensuring convergence within a fixed time. However, this method requires the use of the global gradient norm (calculated based on the gradients of all model parameters) when determining the update amount of model parameters. The update amount of each model parameter is affected by other model parameters, meaning that the updates of all model parameters are forcibly coupled and cannot be updated independently. When the gradients of model parameters differ significantly, the model parameter with the smaller gradient is not updated sufficiently, affecting the model optimization effect.

[0005] Therefore, how to reduce the storage pressure faced during model training and improve the model optimization effect is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, this application provides a control method for training an image classification model, the method comprising:

[0007] Obtain the objective function, current model parameters, and training dataset of the model to be trained; the training dataset is images.

[0008] Based on the objective function, determine the gradient components of the current model parameters;

[0009] Based on the gradient components of each current model parameter, the corresponding target first-order momentum is determined independently.

[0010] The target first-order momentum is scaled to obtain the target parameter update amount of the current model parameters;

[0011] The target model parameters are determined based on the current model parameters and the target parameter update amount;

[0012] Under the target model parameters, the model to be trained is iteratively trained using the training dataset until the training termination condition is met to obtain the target model; the target model is used for image processing.

[0013] Optionally, scaling the target first-order momentum to obtain the target parameter update amount of the current model parameters includes:

[0014] Obtain the target hyperparameters; the target hyperparameters include at least a first target power exponent and a second target power exponent.

[0015] The absolute value of the target first-order momentum is raised to the power of the first target to determine the first update magnitude; the absolute value of the target first-order momentum is raised to the power of the second target to determine the second update magnitude.

[0016] Based on the target's first-order momentum, determine the update direction of the target parameter update quantity;

[0017] The target parameter update amount is determined based on the first update magnitude, the second update magnitude, and the update direction.

[0018] Optionally, the target hyperparameter further includes a first target weight coefficient, a second target weight coefficient, and a target learning rate; determining the target parameter update amount based on the first update magnitude, the second update magnitude, and the update direction includes:

[0019] Based on the first update magnitude and the update direction, determine the far-end accelerated update amount; based on the second update magnitude and the update direction, determine the near-end convergent update amount.

[0020] Based on the first target weight coefficient and the second target weight coefficient, the weighted sum of the far-end accelerated update amount and the near-end convergent update amount is obtained to obtain the initial parameter update amount.

[0021] Based on the target learning rate, the initial parameter update amount is scaled to obtain the target parameter update amount.

[0022] Optionally, obtain the target hyperparameters, including:

[0023] Obtain a pre-constructed mapping relationship; the mapping relationship is the correspondence between the network layers, iteration number and hyperparameters of the model to be trained; the hyperparameters include a first power exponent, a second power exponent, a first weight coefficient, a second weight coefficient and a learning rate;

[0024] Obtain the target network layer corresponding to the current model parameters, and the current iteration number;

[0025] Based on the mapping relationship, the target hyperparameters corresponding to the target network layer and the current iteration number are determined.

[0026] Optionally, based on the gradient components of each current model parameter, the corresponding target first-order momentum is determined independently, including:

[0027] Obtain the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameter itself;

[0028] The gradient weights corresponding to the gradient components are determined based on the momentum decay coefficient.

[0029] The target first-order momentum is obtained by weighted summation of the current first-order momentum and the gradient components based on the momentum decay coefficient and the gradient weight.

[0030] Optionally, based on the gradient components of each current model parameter, the corresponding target first-order momentum is determined independently, including:

[0031] Obtain the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameter itself;

[0032] The current first-order momentum is scaled according to the momentum decay coefficient;

[0033] The target first-order momentum is determined based on the momentum obtained by scaling the current first-order momentum and the gradient component.

[0034] Optionally, after obtaining the objective function of the model to be trained, the following may also be included:

[0035] Determine whether the objective function satisfies the objective condition; the objective condition is an L-smoothness condition or a PL condition;

[0036] If the objective function is satisfied, proceed to the step of determining the gradient components of the current model parameters based on the objective function.

[0037] This application also provides a control device for training an image classification model, the device comprising:

[0038] The parameter acquisition module is used to acquire the objective function, current model parameters, and training dataset of the model to be trained; the training dataset is images.

[0039] The gradient determination module is used to determine the gradient components of the current model parameters based on the objective function.

[0040] The momentum determination module is used to independently determine the target first-order momentum of each of the current model parameters based on its own gradient components.

[0041] The update amount determination module is used to scale the first-order momentum of the target to obtain the target parameter update amount of the current model parameters;

[0042] The parameter determination module is used to determine the target model parameters based on the current model parameters and the target parameter update amount;

[0043] The training module is used to iteratively train the model to be trained using the training dataset under the target model parameters until the training termination condition is met to obtain the target model; the target model is used for image processing.

[0044] This application also provides an electronic device, including:

[0045] Memory, used to store computer programs;

[0046] A processor is configured to implement the steps of the control method for training any of the above-described image classification models when executing the computer program.

[0047] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the control method for training any of the above-described image classification models.

[0048] In summary, this application provides a control method, apparatus, device, and medium for training an image classification model. The process involves obtaining the objective function, current model parameters, and training dataset of the model to be trained; determining the gradient components of the current model parameters based on the objective function; independently determining the corresponding first-order target momentum based on the gradient components of each current model parameter; scaling the first-order target momentum to obtain the target parameter update amount of the current model parameters; determining the target model parameters based on the target parameter update amount and the current model parameters; and iteratively training the model to be trained using the training dataset under the target model parameters until the target model is obtained. The training process only requires storing the first-order target momentum and model parameters, saving storage resources and reducing training costs. The update of each model parameter depends only on its own gradient component and is independent of other model parameters, achieving independent updates of each model parameter, ensuring sufficient model parameter updates, and improving model optimization performance. Attached Figure Description

[0049] Figure 1 A flowchart illustrating the control method for training the image classification model provided in this application;

[0050] Figure 2 A flowchart illustrating the process for determining the target parameter update amount provided in this application;

[0051] Figure 3 The first verification accuracy curve provided for this application;

[0052] Figure 4 The second verification accuracy curve provided for this application;

[0053] Figure 5 The third verification accuracy curve provided for this application;

[0054] Figure 6 The fourth verification accuracy curve provided for this application;

[0055] Figure 7 The fifth verification accuracy curve provided for this application;

[0056] Figure 8 The sixth verification accuracy curve provided for this application. Detailed Implementation

[0057] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0058] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0059] Please refer to Figure 1 , Figure 1 A flowchart illustrating the control method for training the image classification model provided in this application is shown. The method includes:

[0060] S1: Obtain the objective function, current model parameters, and training dataset of the model to be trained; the training dataset consists of images.

[0061] In this application, a pre-constructed training model, an objective function for measuring model performance and guiding model parameter updates, and a training dataset for training the training model are provided. The training model is used for image processing (e.g., image classification), and the training dataset consists of images.

[0062] This application does not impose any specific restrictions on the architecture of the model to be trained. For example, the model to be trained may be an image classification model such as ResNet-34 (Residual Network-34), VGG-11 (Visual Geometry Group-11), or DenseNet-121 (Densely Connected Network-121).

[0063] The training datasets mentioned above can be selected according to actual needs, and this application does not impose any special restrictions. For example, the training datasets can be CIFAR-10 (Canadian Institute for Advanced Research-10) or CIFAR-100 (Canadian Institute for Advanced Research-100).

[0064] The objective function described above can also be set according to actual needs, and this application does not impose any special restrictions on it.

[0065] This application also obtains the current model parameters, that is, the model parameters of the model to be trained during this iteration of training. The model parameters include the learnable parameters that participate in gradient updates during iterative training. Obtaining the current model parameters is crucial for determining the target model parameters for the model to be trained in the next iteration.

[0066] S2: Determine the gradient components of the current model parameters based on the objective function.

[0067] As an optional implementation, the gradient components of the current model parameters are determined based on the partial derivatives of the objective function with respect to the current model parameters. These gradient components then guide the updating of the current model parameters, thus enabling the updating of the current model parameters.

[0068] S3: Determine the target first-order momentum independently based on the gradient components of each current model parameter.

[0069] S4: Scale the first-order momentum of the target to obtain the target parameter update amount of the current model parameters.

[0070] Understandably, for large-scale models to be trained, the amount of data for model parameters is enormous. When training and optimizing these models, in addition to storing the model parameters, it's also necessary to store other parameters involved in the training and optimization process (such as the first and second momentum used by the Adam or AdamW optimizer), further increasing storage pressure and making training extremely costly. Furthermore, if the update of the model parameters themselves is affected by other model parameters, it can easily lead to insufficient updates of the model parameters, affecting the model optimization effect.

[0071] Therefore, in this application, for each current model parameter, the target first-order momentum corresponding to that current model parameter is independently determined based on its own gradient component. The target first-order momentum can characterize the "direction" or "inertial tendency" of the current model parameter in the parameter space, laying the foundation for determining the target parameter update amount.

[0072] This application scales the target first-order momentum to obtain the target parameter update amount of the current model parameters. The scaling mentioned above includes, but is not limited to, exponential scaling and linear scaling, etc., which this application does not specifically limit. The method of determining the target parameter update amount by scaling the target first-order momentum utilizes the direction and kinetic energy provided by the gradient component, and also achieves adaptive adjustment of the target parameter update amount through scaling (when the gradient component is large, the target parameter update amount obtained by scaling the target first-order momentum is relatively large; when the gradient component is small, the target parameter update amount is relatively small), thereby accelerating the convergence of the objective function; smoothing noise and suppressing oscillations, making the convergence path smoother; and also enabling the crossing of local minima, avoiding getting trapped in local optima. In short, it makes the model training and optimization process more efficient and stable.

[0073] The specific implementation methods for determining the target's first-order momentum based on the gradient components and for determining the target parameter update amount based on the target's first-order momentum will be introduced in subsequent embodiments, and will not be elaborated here.

[0074] On the one hand, this application only needs to determine the target first-order momentum, reducing the parameters used in the model training and optimization process, and lowering storage pressure and training costs. On the other hand, the target parameter update amount of the current model parameter depends only on its own target first-order momentum; the target first-order momentum, in turn, depends only on its own gradient components and is independent of the gradient components of other current model parameters; this ensures that the target parameter update amount of each current model parameter can be determined independently, that is, to achieve independent updates of each model parameter, ensuring sufficient updates of model parameters and improving model optimization performance.

[0075] S5: Determine the target model parameters based on the current model parameters and the target parameter update amount.

[0076] Based on the above, the target model parameters can be determined according to the current model parameters and the target parameter update amount, which is also the model parameters for the next iteration of training. This application does not specifically limit the method for determining the target model parameters. As an optional embodiment, the target model parameters are obtained by subtracting the target parameter update amount from the current model parameters.

[0077] S6: Under the target model parameters, use the training dataset to iteratively train the model to be trained until the training termination condition is met to obtain the target model; the target model is used for image processing.

[0078] After determining the target model parameters, the model parameters of the model to be trained are updated to match the target model parameters. Using the training dataset, a new round of training is performed on the model to be trained, and it is determined whether the training termination condition is met. If the training termination condition is not met, the above steps are repeated to determine the target model parameters for the next round of training. This process is iterated until the training termination condition is met. The training termination condition can be reaching a preset number of training iterations or the objective function converging; this application does not impose any particular limitation on this.

[0079] Therefore, the image classification model training control method provided in this application independently determines the corresponding target first-order momentum based on the gradient components of each current model parameter. The target first-order momentum is scaled to obtain the target parameter update for the current model parameters. The target model parameters are determined based on the current model parameters and the target parameter update. During model training and optimization, only the target first-order momentum and model parameters need to be stored, saving storage resources and reducing training costs. Furthermore, the update of model parameters depends only on their own gradient components and is independent of other model parameters, achieving independent updates for each model parameter, ensuring sufficient model parameter updates, and improving model optimization performance.

[0080] Based on the above embodiments:

[0081] The following section details the specific process for determining the target parameter update amount.

[0082] As an optional embodiment, scaling the target first-order momentum to obtain the target parameter update amount of the current model parameters includes:

[0083] Obtain the target hyperparameters; the target hyperparameters include at least the first target power exponent and the second target power exponent;

[0084] For the absolute value of the target first-order momentum, perform a power operation with the exponent being the first target power exponent to determine the first update magnitude; for the absolute value of the target first-order momentum, perform a power operation with the exponent being the second target power exponent to determine the second update magnitude.

[0085] Determine the update direction of the target parameter update based on the target's first-order momentum;

[0086] The target parameter update amount is determined based on the first update magnitude, the second update magnitude, and the update direction.

[0087] In this embodiment, a first target power exponent and a second target power exponent are first obtained, and then the target first-order momentum is exponentially scaled using the target power exponent. The values ​​of the first target power exponent and the second target power exponent can be different, and this embodiment does not impose any particular limitation on their specific values.

[0088] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the process of determining the target parameter update amount provided in this application. In this embodiment, the absolute value of the target first-order momentum is first taken, and then the absolute value of the target first-order momentum is exponentially scaled by the first and second target power exponents to obtain the first update amplitude and the second update amplitude. On the one hand, exponentializing the absolute value of the target first-order momentum avoids the generation of complex numbers or sign reversals in the calculation result, i.e., the update amplitude. On the other hand, converting the absolute value of the target first-order momentum into the update amplitude of the target parameter update amount enables adaptive adjustment of the update amplitude (the larger the absolute value of the target first-order momentum, the larger the update amplitude; the smaller the absolute value of the target first-order momentum, the smaller the update amplitude), accelerating model training and optimization.

[0089] In this embodiment, the update direction of the target parameter update is determined based on the target first-order momentum. The target first-order momentum is determined based on the gradient components of the current model parameters. Therefore, essentially, the update direction of the target parameter update is determined based on the gradient components of the current model parameters. This approach ensures that the update direction of the target parameter update is consistent with the direction of the gradient components, achieving robust updates.

[0090] Finally, based on the update magnitude and update direction mentioned above, the target parameter update amount can be determined.

[0091] As an optional embodiment, the target hyperparameters further include a first target weight coefficient, a second target weight coefficient, and a target learning rate; the target parameter update amount is determined based on the first update magnitude, the second update magnitude, and the update direction, including:

[0092] Based on the first update magnitude and update direction, determine the far-end accelerated update amount; based on the second update magnitude and update direction, determine the near-end convergent update amount.

[0093] Based on the weight coefficients of the first objective and the weight coefficients of the second objective, the weighted sum of the far-end accelerated update and the near-end convergent update is obtained to get the initial parameter update.

[0094] Based on the target learning rate, the initial parameter update amount is scaled to obtain the target parameter update amount.

[0095] Based on the determination of the first and second update amplitudes, the update amount can be determined by combining the update amplitude and update direction. As an optional embodiment, if the target first-order momentum is positive, the update direction of the target parameter update amount is positive; if the target first-order momentum is negative, the update direction is negative. Based on this, the target first-order momentum is substituted into the sign function for calculation. If the target first-order momentum is positive, the calculation result is 1; if the target first-order momentum is zero, the calculation result is 0; if the target first-order momentum is negative, the calculation result is -1. The product of the calculation result and the first update amplitude is used as the far-end acceleration update amount. The product of the calculation result and the second update amplitude is used as the near-end convergence update amount.

[0096] like Figure 2 As shown, in this embodiment, a first target weight coefficient, a second target weight coefficient, and a target learning rate are also obtained. The target weight coefficients are used to perform a weighted summation of the above-mentioned update amounts, and the target learning rate is used to determine the final target parameter update amount. The first target weight coefficient, the second target weight coefficient, and the target learning rate can all be set according to actual needs, and this embodiment does not impose any special limitations.

[0097] The method used in this embodiment, which involves weighted summation of the far-end acceleration and near-end convergence to determine the target parameter update, can accelerate model parameter updates when the current model parameters are far from the optimal solution, and avoid oscillations and stably approach the optimal solution when the current model parameters are close to the optimal solution.

[0098] Based on the above, the calculation of the target parameter update can be expressed by the following formula (1):

[0099] (1)

[0100] in, For the target parameter update amount, For the target learning rate, The first objective weighting coefficient, The second objective weighting coefficient, For the first objective power exponent, 2- The second objective power exponent. For symbolic functions, For the target first momentum, This represents the current iteration number.

[0101] The above , , , as well as All settings can be customized according to actual needs. For example, , , Scaling linearly or sublinearly with batch size. Furthermore, when... When the objective function converges in a fixed time, the upper bound of the convergence time is independent of the initial conditions, meaning the upper bound of the convergence time is independent of the initial model parameters of the model being trained. When the objective function converges in a finite time, the upper bound of the convergence time is related to the initial model parameters.

[0102] In addition, when and When the above formula (1) degenerates into a symbolic optimization algorithm (including SignSGD and Signum), it provides a theoretical basis for the two and fills the theoretical gap.

[0103] Correspondingly, the target model parameters can be calculated using the following formula:

[0104] (2)

[0105] in, For the target model parameters, These are the current model parameters. This represents the target parameter update amount.

[0106] In summary, in this embodiment, the target first-order momentum is used as the basis for determining the update direction of the target parameter update, and the update magnitude of the target parameter update is determined by power-weighted method, which can improve the convergence speed, convergence smoothness and convergence reliability, and further optimize the model training process.

[0107] As an optional embodiment, obtaining the target hyperparameters includes:

[0108] Obtain the pre-constructed mapping relationship; the mapping relationship is the correspondence between the network layers, iteration number and hyperparameters of the model to be trained; the hyperparameters include the first power exponent, the second power exponent, the first weight coefficient, the second weight coefficient and the learning rate;

[0109] Obtain the target network layer corresponding to the current model parameters, and the current iteration number;

[0110] Based on the mapping relationship, the target network layer and the target hyperparameters corresponding to the current iteration number are determined.

[0111] Considering the varying levels of feature abstraction, gradient magnitude, and noise in different network layers of the model under training, the requirements for hyperparameters also differ. For example, shallow networks are responsible for capturing general image features, with large gradient magnitudes and low noise. Therefore, a larger learning rate can be used to accelerate feature learning, and a smaller second power exponent can be used to achieve fixed-time convergence. Mid-level networks progressively increase the level of feature abstraction, while gradually decreasing the gradient magnitude, and are at risk of gradient vanishing or exploding. Therefore, a smaller learning rate and a larger second power exponent can be used to smoothly transition from fixed-time convergence to finite-time convergence. Deep networks have highly abstract features, and their gradients are often very small and extremely sensitive to subtle changes. Therefore, the learning rate can be further reduced, and the second power exponent increased, allowing the parameter update magnitude to decay rapidly with momentum, achieving fine-grained convergence.

[0112] This embodiment also considers that the training objective changes with the training phase. For example, in the early stages of model training, the training objective is to prevent gradient explosion and achieve momentum accumulation, so a larger learning rate and a smaller second power exponent can be used to achieve fixed-time convergence. In the acceleration phase of model training, the training objective is to quickly reduce errors and cross flat regions, so the learning rate can be reduced and the second power exponent increased to smoothly transition from fixed-time convergence to finite-time convergence. In the fine-tuning phase of model training, the training objective is to achieve fine convergence and improve final accuracy, so the learning rate can be further reduced and the second power exponent increased to make the parameter update magnitude decay rapidly with momentum.

[0113] Based on the above considerations, this embodiment pre-sets corresponding hyperparameters for different network layers and iteration numbers, thus constructing a mapping relationship. Specifically, the mapping relationship is the correspondence between the network layers, iteration numbers, and hyperparameters of the model to be trained. The specific values ​​of the hyperparameters corresponding to each network layer and iteration number can be set according to actual needs; this embodiment does not impose any particular limitations on this.

[0114] When obtaining the target hyperparameters, first obtain the target network layer corresponding to the current model parameters and the current iteration number. Then, based on the mapping relationship, determine the target hyperparameters corresponding to the target network layer and the current iteration number. When the target network layer and the current iteration number corresponding to the current model parameters are different, the hyperparameters used to update the current model parameters also change accordingly, thereby maximizing the efficiency of model optimization training.

[0115] As an optional implementation, the target first-order momentum is independently determined based on the gradient components of each current model parameter, including:

[0116] Obtain the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameters;

[0117] Determine the gradient weights corresponding to the gradient components based on the momentum decay coefficient.

[0118] Based on the momentum decay coefficient and gradient weight, the current first-order momentum and gradient components are weighted and summed to obtain the target first-order momentum.

[0119] In this embodiment, the momentum decay coefficient and the current first-order momentum are first obtained. The momentum decay coefficient is pre-set, and its value can be set according to actual needs; this embodiment does not impose any particular limitation on it. The current first-order momentum is determined based on the historical gradient components corresponding to the current model parameters. The momentum decay coefficient is equivalent to the weight corresponding to the current first-order momentum, determining the degree of influence of the historical gradient components on the target first-order momentum.

[0120] The gradient weights mentioned above are determined based on the momentum decay coefficient. As an optional embodiment, the difference between the gradient weight and the momentum decay coefficient is used as the gradient weight. The gradient weight is equivalent to the weight of the current gradient component, determining the degree of influence of the current gradient component on the target's first-order momentum.

[0121] The target first-order momentum is obtained by weighted summation of the current first-order momentum and gradient components based on the momentum decay coefficient and gradient weights. The target first-order momentum can be calculated using the following formula (3):

[0122] , (3)

[0123] in, For the target first momentum, Given the current first-order momentum, The momentum decay coefficient, For gradient components, This represents the current number of training iterations. The value can be set to 0.9.

[0124] As can be seen, the target first-order momentum in this embodiment depends not only on the current gradient component but also on the historical gradient components. Therefore, even if the current gradient component changes abruptly due to noise from a small batch of samples, the smoothness of the target first-order momentum can be ensured through the influence of historical gradient components, thereby reducing oscillations during model parameter updates and making the model training and optimization process more stable, especially suitable for large-scale models to be trained.

[0125] As an optional implementation, the target first-order momentum is independently determined based on the gradient components of each current model parameter, including:

[0126] Obtain the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameters;

[0127] The current first-order momentum is scaled according to the momentum decay coefficient;

[0128] The target first-order momentum is determined based on the momentum obtained by scaling the current first-order momentum and the gradient components.

[0129] In this embodiment, the obtained momentum decay coefficient is also pre-set, and its value can be set according to actual needs. This embodiment does not impose any special limitations on this. The current first-order momentum is also determined based on the historical gradient components corresponding to the current model parameters.

[0130] In this embodiment, the current first-order momentum is scaled using the momentum decay coefficient; and the target first-order momentum is determined based on the scaled momentum and the gradient components. The target first-order momentum can be calculated using the following formula (4):

[0131] , (4)

[0132] in, For the target first momentum, Given the current first-order momentum, The momentum decay coefficient, For gradient components, This represents the current number of training iterations. The value can be set to 0.9.

[0133] As can be seen, this embodiment does not use gradient weights to scale the current gradient components. Instead, it directly accumulates the current gradient components into the target first-order momentum, achieving non-normalized accumulation of historical gradient components and emphasizing long-term trends. As model training progresses, the gradient components gradually accumulate. Even in the later stages of model training, when the gradient components become smaller, the accumulated momentum remains relatively large. Therefore, the target first-order momentum remains large, providing stronger convergence momentum and accelerating the model training and optimization process.

[0134] In addition, the gradient components of the current model parameters can be directly used as the target first-order momentum to simplify the calculation, thus enabling model parameter updates even under resource constraints.

[0135] Based on the above embodiments, ResNet-34, VGG-11, and DenseNet-121 were used sequentially as models to be trained, and CIFAR-10 and CIFAR-100 were used sequentially as training datasets. The accuracy of the image classification model training control method provided in this application and various existing control methods was verified on the training datasets. The model training configuration was as follows: batch size of 128, training for 200 epochs, and the learning rate decreased by 0.1 times at the 150th epoch.

[0136] Table 1 is a comparison table of the first accuracy of each model to be trained under different control methods. The training dataset corresponding to Table 1 is CIFAR-10. As shown in Table 1, the models to be trained include ResNet-34, VGG-11 and DenseNet-121; the control methods include the existing SGD, SGDM, AdamW, SignSGD, Signum, and the control method for training the image classification model provided in this application. According to the different calculation methods of the first-order momentum of the target, the control method for training the image classification model provided in this application is further divided into: the first method (directly taking the gradient component of the current model parameter as the first-order momentum of the target), the second method (determining the first-order momentum of the target based on formula (3)) and the third method (determining the first-order momentum of the target based on formula (4)).

[0137] It should also be noted that in the third method described above, the target learning rate is set to 0.01, the first target power exponent is set to 1.4, the second target power exponent is set to 0.6, and both the first and second target weight coefficients are set to 0.5. In the first method, the momentum decay coefficient is set to 0.9, and the settings of other target hyperparameters are consistent with those in the third method. In the second method, the momentum decay coefficient is set to 0.9, the first target power exponent is set to 1.2, the second target power exponent is set to 0.8, and the settings of other target hyperparameters are consistent with those in the third method.

[0138] Table 1. Comparison of the first accuracy of each model to be trained under different control methods.

[0139]

[0140] Table 2 shows a comparison of the second accuracy of each model to be trained under different control methods. The training dataset corresponding to Table 2 is CIFAR-100. As shown in Table 2, the models to be trained include ResNet-34, VGG-11, and DenseNet-121; the control methods include existing SGD, SGDM, AdamW, SignSGD, and Signum, as well as the first, second, and third methods provided in this application.

[0141] Table 2 Comparison of the second accuracy of each model to be trained under different control methods

[0142]

[0143] Based on Tables 1 and 2, it is evident that the accuracy of the first, second, and third methods provided in this application is significantly higher than that of existing control methods.

[0144] This application also addresses the scenario where the training dataset is CIFAR-10 and the models to be trained are ResNet-34, VGG-11, and DenseNet-121, determining the variation of accuracy for each control method with the number of training iterations. Please refer to [link / reference needed]. Figures 3 to 5 .in, Figure 3 The first verification accuracy curve provided for this application corresponds to the VGG-11 model to be trained. Figure 4 The second verification accuracy curve provided for this application corresponds to the DenseNet-121 model to be trained; Figure 5 The third verification accuracy curve provided in this application corresponds to the ResNet-34 model to be trained. It can be seen that the accuracy of the first, second, and third methods provided in this application is significantly higher than that of existing control methods, with the third method exhibiting the highest accuracy.

[0145] This application also addresses the scenario where the training dataset is CIFAR-100 and the models to be trained are ResNet-34, VGG-11, and DenseNet-121, determining the variation of accuracy for each control method with the number of training iterations. Please refer to [link / reference]. Figures 6 to 8 .in, Figure 6 The fourth verification accuracy curve provided for this application corresponds to the VGG-11 model to be trained. Figure 7 The fifth verification accuracy curve provided for this application corresponds to the DenseNet-121 model to be trained. Figure 8 The sixth verification accuracy curve provided in this application corresponds to the ResNet-34 model to be trained. It can be seen that the accuracy of the first, second, and third methods provided in this application is significantly higher than that of existing control methods, with the third method exhibiting the highest accuracy.

[0146] As an optional embodiment, after obtaining the objective function of the model to be trained, the method further includes:

[0147] Determine whether the objective function satisfies the objective condition; the objective condition is either an L-smoothness condition or a PL condition.

[0148] If satisfied, proceed to the step of determining the gradient components of the current model parameters based on the objective function.

[0149] In this embodiment, after obtaining the objective function of the model to be trained, it is first determined whether the objective function satisfies the L-smoothness condition or the PL condition. If the objective function satisfies the L-smoothness condition or the PL condition, it provides a theoretical basis for setting the upper limit of the target learning rate and ensures convergence within a finite or fixed time. Therefore, after determining that the objective function satisfies the target condition, the step of determining the gradient components of the current model parameters based on the objective function is then performed to ensure the effectiveness of model training and optimization.

[0150] The L-smoothness condition can be expressed by the following formula (5):

[0151] (5)

[0152] in, One of the current model parameters of the model to be trained. For another current model parameter of the model to be trained, For the objective function pair gradient components, For the objective function pair gradient components, It is a smoothness constant. Let be the Euclidean norm between the two model parameters.

[0153] The PL condition can be expressed by the following formula (6):

[0154] (6)

[0155] in, These are the current model parameters of the model to be trained. The parameters are globally optimal. For the objective function pair gradient components, Let be the square of the Euclidean norm of the gradient component. PL constant, The loss corresponding to the current model parameters. This represents the minimum loss corresponding to the globally optimal parameters.

[0156] In summary, in this embodiment, the step of determining the gradient components of the current model parameters is only taken after the objective function is determined to meet the objective conditions. This ensures convergence and thus guarantees the effectiveness of model training and optimization.

[0157] This application also provides a control device for training an image classification model, the device comprising:

[0158] The parameter acquisition module is used to obtain the objective function, current model parameters, and training dataset of the model to be trained; the training dataset consists of images.

[0159] The gradient determination module is used to determine the gradient components of the current model parameters based on the objective function.

[0160] The momentum determination module is used to independently determine the target first-order momentum of each current model parameter based on its own gradient components.

[0161] The update amount determination module is used to scale the first-order momentum of the target to obtain the target parameter update amount of the current model parameters;

[0162] The parameter determination module is used to determine the target model parameters based on the current model parameters and the target parameter update amount;

[0163] The training module is used to iteratively train the model to be trained using the training dataset under the parameters of the target model until the training termination condition is met to obtain the target model; the target model is used for image processing.

[0164] For a detailed description of the control device for training the image classification model provided in this application, please refer to the embodiments of the control method for training the image classification model; this application will not elaborate further here.

[0165] Based on the above embodiments:

[0166] As an optional embodiment, the update amount determination module includes:

[0167] The hyperparameter acquisition module is used to acquire the target hyperparameters; the target hyperparameters include at least a first target power exponent and a second target power exponent.

[0168] The amplitude determination module is used to perform a power operation on the absolute value of the target first-order momentum with the exponent being the first target power exponent to determine the first update amplitude; and to perform a power operation on the absolute value of the target first-order momentum with the exponent being the second target power exponent to determine the second update amplitude.

[0169] The direction determination module is used to determine the update direction of the target parameter update quantity based on the target's first-order momentum.

[0170] The first update submodule is used to determine the target parameter update amount based on the first update magnitude, the second update magnitude, and the update direction.

[0171] As an optional embodiment, the target hyperparameters further include a first target weight coefficient, a second target weight coefficient, and a target learning rate; the update determination submodule includes:

[0172] The second update submodule is used to determine the far-end accelerated update amount based on the first update magnitude and update direction; and to determine the near-end convergent update amount based on the second update magnitude and update direction.

[0173] The initial determination module is used to perform a weighted summation of the far-end accelerated update amount and the near-end convergent update amount based on the first target weight coefficient and the second target weight coefficient to obtain the initial parameter update amount.

[0174] The target determination module is used to scale the initial parameter update amount based on the target learning rate to obtain the target parameter update amount.

[0175] As an optional embodiment, the hyperparameter acquisition module includes:

[0176] The mapping relationship determination module is used to obtain the pre-constructed mapping relationship; the mapping relationship is the correspondence between the network layers, iteration number and hyperparameters of the model to be trained; the hyperparameters include the first power exponent, the second power exponent, the first weight coefficient, the second weight coefficient and the learning rate;

[0177] The first information acquisition module is used to acquire the target network layer corresponding to the current model parameters, as well as the current iteration number;

[0178] The hyperparameter determination module is used to determine the target hyperparameters corresponding to the target network layer and the current iteration number based on the mapping relationship.

[0179] As an optional embodiment, the momentum determination module includes:

[0180] The second information acquisition module is used to acquire the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameters themselves.

[0181] The weight determination module is used to determine the gradient weights corresponding to the gradient components based on the momentum decay coefficient.

[0182] The first momentum determination submodule is used to perform a weighted summation of the current first-order momentum and gradient components based on the momentum decay coefficient and gradient weights to obtain the target first-order momentum.

[0183] As an optional embodiment, the momentum determination module includes:

[0184] The third information acquisition module is used to acquire the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameters themselves.

[0185] The momentum scaling module is used to scale the current first-order momentum according to the momentum decay coefficient.

[0186] The second momentum determination submodule is used to determine the target first-order momentum based on the momentum obtained by scaling the current first-order momentum and the gradient components.

[0187] As an optional embodiment, the device further includes:

[0188] The condition determination module is used to determine whether the objective function of the model to be trained satisfies the objective condition after obtaining the objective function; the objective condition is either the L-smoothness condition or the PL condition; if it is satisfied, the gradient determination module is triggered.

[0189] This application also provides an electronic device, including:

[0190] Memory, used to store computer programs;

[0191] A processor is used to execute computer programs to control the training of any of the above-mentioned image classification models.

[0192] For a detailed description of the electronic device provided in this application, please refer to the embodiments of the control method for image classification model training; this application will not elaborate further here.

[0193] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the control method for training any of the above-mentioned image classification models.

[0194] The aforementioned storage media include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or integrated into dedicated logic circuitry.

[0195] For a detailed description of the storage medium provided in this application, please refer to the embodiments of the control method for training the image classification model; this application will not elaborate further here.

[0196] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0197] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A control method for training an image classification model, characterized in that, The method includes: Obtain the objective function, current model parameters, and training dataset of the model to be trained; the training dataset is images. Based on the objective function, determine the gradient components of the current model parameters; Based on the gradient components of each current model parameter, the corresponding target first-order momentum is determined independently. The target first-order momentum is scaled to obtain the target parameter update amount of the current model parameters; The target model parameters are determined based on the current model parameters and the target parameter update amount; Under the target model parameters, the model to be trained is iteratively trained using the training dataset until the training termination condition is met to obtain the target model; the target model is used for image processing. Scaling the target first-order momentum to obtain the target parameter update amount of the current model parameters includes: obtaining target hyperparameters; the target hyperparameters include at least a first target power exponent and a second target power exponent; raising the absolute value of the target first-order momentum to the power of the first target power exponent to determine a first update magnitude; raising the absolute value of the target first-order momentum to the power of the second target power exponent to determine a second update magnitude; determining the update direction of the target parameter update amount based on the target first-order momentum; and determining the target parameter update amount based on the first update magnitude, the second update magnitude, and the update direction. The target hyperparameters further include a first target weight coefficient, a second target weight coefficient, and a target learning rate. Determining the target parameter update amount based on the first update amplitude, the second update amplitude, and the update direction includes: determining a far-end accelerated update amount based on the first update amplitude and the update direction; determining a near-end convergent update amount based on the second update amplitude and the update direction; performing a weighted summation of the far-end accelerated update amount and the near-end convergent update amount based on the first target weight coefficient and the second target weight coefficient to obtain an initial parameter update amount; and scaling the initial parameter update amount based on the target learning rate to obtain the target parameter update amount.

2. The control method for training the image classification model as described in claim 1, characterized in that, Obtain the target hyperparameters, including: Obtain a pre-constructed mapping relationship; the mapping relationship is the correspondence between the network layers, iteration number and hyperparameters of the model to be trained; the hyperparameters include a first power exponent, a second power exponent, a first weight coefficient, a second weight coefficient and a learning rate; Obtain the target network layer corresponding to the current model parameters, and the current iteration number; Based on the mapping relationship, the target hyperparameters corresponding to the target network layer and the current iteration number are determined.

3. The control method for training the image classification model as described in claim 1, characterized in that, Based on the gradient components of each current model parameter, independently determine the corresponding target first-order momentum, including: Obtain the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameter itself; The gradient weights corresponding to the gradient components are determined based on the momentum decay coefficient. The target first-order momentum is obtained by weighted summation of the current first-order momentum and the gradient components based on the momentum decay coefficient and the gradient weight.

4. The control method for training the image classification model as described in claim 1, characterized in that, Based on the gradient components of each current model parameter, independently determine the corresponding target first-order momentum, including: Obtain the pre-set momentum decay coefficient and the current first-order momentum corresponding to the current model parameter itself; The current first-order momentum is scaled according to the momentum decay coefficient; The target first-order momentum is determined based on the momentum obtained by scaling the current first-order momentum and the gradient component.

5. The control method for training the image classification model as described in claim 1, characterized in that, After obtaining the objective function of the model to be trained, the following steps are also included: Determine whether the objective function satisfies the objective condition; the objective condition is an L-smoothness condition or a PL condition; If the objective function is satisfied, proceed to the step of determining the gradient components of the current model parameters based on the objective function.

6. A control device for training an image classification model, characterized in that, The device includes: The parameter acquisition module is used to acquire the objective function, current model parameters, and training dataset of the model to be trained; the training dataset is images. The gradient determination module is used to determine the gradient components of the current model parameters based on the objective function. The momentum determination module is used to independently determine the target first-order momentum of each of the current model parameters based on its own gradient components. The update amount determination module is used to scale the first-order momentum of the target to obtain the target parameter update amount of the current model parameters; The parameter determination module is used to determine the target model parameters based on the current model parameters and the target parameter update amount; The training module is used to iteratively train the model to be trained using the training dataset under the target model parameters until the training termination condition is met to obtain the target model; the target model is used for image processing. The update quantity determination module includes: The hyperparameter acquisition module is used to acquire the target hyperparameters; the target hyperparameters include at least a first target power exponent and a second target power exponent. The amplitude determination module is used to perform a power operation on the absolute value of the target first-order momentum with the exponent being the first target power exponent to determine the first update amplitude; and to perform a power operation on the absolute value of the target first-order momentum with the exponent being the second target power exponent to determine the second update amplitude. The direction determination module is used to determine the update direction of the target parameter update quantity based on the target's first-order momentum. The first update submodule is used to determine the target parameter update amount based on the first update magnitude, the second update magnitude, and the update direction; The target hyperparameters also include the first target weight coefficient, the second target weight coefficient, and the target learning rate; the first update submodule includes: The second update submodule is used to determine the far-end accelerated update amount based on the first update magnitude and update direction; and to determine the near-end convergent update amount based on the second update magnitude and update direction. The initial determination module is used to perform a weighted summation of the far-end accelerated update amount and the near-end convergent update amount based on the first target weight coefficient and the second target weight coefficient to obtain the initial parameter update amount. The target determination module is used to scale the initial parameter update amount based on the target learning rate to obtain the target parameter update amount.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to implement the steps of the control method for training an image classification model as described in any one of claims 1 to 5 when executing the computer program.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the control method for training an image classification model as described in any one of claims 1 to 5.

Images