Loss function optimization method of image detection model
By adding regularization terms to the loss function and dynamically adjusting the weight decay parameters, the problem of overfitting of neural network models is solved, and the generalization ability and test set performance of the model are improved.
Patent Information
- Application Number
- CN202411629345.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-03-25
AI Technical Summary
In the prior art, neural network models are prone to overfitting problems, resulting in good performance on the training set but poor performance on the test set.
Optimize regularization parameters to reduce the risk of overfitting by adding regularization terms in the loss function, initializing and dynamically adjusting the weight decay parameters for each weight parameter.
It effectively reduces the complexity of the model, improves the generalization ability of the model, reduces the risk of overfitting, and improves the performance of the model on the test set.
Smart Images
Figure CN119693736A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of training neural network models, and in particular to a loss function optimization method for an image detection model. Background Art
[0002] With the continuous development of deep learning and neural network technology, neural network models have achieved remarkable results in various application fields. However, due to the high complexity and number of parameters of neural network models, overfitting problems are prone to occur, that is, the model performs well on the training set but poorly on the test set. Summary of the invention
[0003] The present invention mainly aims to solve the technical problem of prone to overfitting in the prior art and provides a loss function optimization method for an image detection model.
[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a method for optimizing the loss function of an image detection model, which uses the following method to correct the loss function during model training:
[0005] S1. Add a regularization term to the loss function and initialize the weight decay parameter of each weight parameter;
[0006] S2. After each iteration or a certain number of iterations, dynamically adjust the weight decay parameter corresponding to the weight parameter according to the importance of the weight parameter and the level at which it is located;
[0007] S3, applying the dynamically adjusted weight decay parameter to the loss function and calculating the regularization term;
[0008] S4, looping steps S2 and S3, continuously optimizing the regularization parameter during the entire training process, and obtaining a set of regularization parameter candidate values; machine learning will obtain a set of parameters with different effects, some of which may have better recall, some may have better accuracy, and some may have better AP;
[0009] S5. Evaluate the performance of the model on the validation set and select the best regularization parameter configuration.
[0010] Preferably, in step S1, the loss function after adding the regularization term is:
[0011]
[0012] m is the number of samples in the model training data, that is, the number of samples in the data set; y i is the true value of the i-th sample, is the predicted value of the i-th sample, L is the total number of network layers of the image detection model, N l is the total number of weight parameters of the lth layer, wli is the i-th weight parameter of the l-th layer, γ li is the i-th weight parameter w of the l-th layer li The weight decay parameter.
[0013] Before training begins, weight parameters are usually randomly initialized because random initialization can break symmetry and help the model learn different features. During model training, weight parameters are updated through the back propagation algorithm. The back propagation algorithm calculates the gradient of the loss function with respect to the weights, and then uses these gradients to update the weights with the goal of minimizing the loss function. The weight parameters are constantly changing during training, and they are adjusted based on the data and feedback from the loss function. Once the model training is completed, the weight parameters will be fixed and used in the inference phase of the model, at which time the weight parameters no longer change, but are used to predict new input data.
[0014] Preferably, in step S2, the weight attenuation parameter is specifically:
[0015] γ li =α·γ base +β·g li
[0016] In the formula, γ li is the i-th weight parameter w of the l-th layer li The weight decay parameter, α and β are adjustment coefficients, γ base is the basic regularization parameter, g li is the weight parameter w li The evaluation value of γ base Generally, empirical settings are used. Generally, an initial importance assessment value is set for the weight parameter based on domain knowledge or previous research experience. The evaluation value of the weight parameter is to control the weight parameter more finely during the training process, and this value itself changes as the training progresses, involving sensitivity analysis of the weight parameter or its role in the model. This scheme can use two methods to determine the evaluation value: (1) The impact of the weight parameter on the output: By analyzing the impact of the weight parameter on the model output, its importance can be evaluated; (2) Gradient size: The absolute value of the gradient of the weight parameter can be used as an indicator of its importance, that is, directly as an evaluation value.
[0017] α and β train the model on the training set, then evaluate the impact of different regularization parameter values on model performance on the validation set, and finally select the regularization parameter that achieves the best performance.
[0018] As a preference, the weight attenuation parameter corresponding to the weight parameter is dynamically adjusted according to the importance of the weight parameter and the level at which it is located, specifically:
[0019] The Hessian matrix is used to evaluate the importance of each weight parameter:
[0020]
[0021] In the formula, H ii is the element in the i-th row and i-th column of the Hessian matrix, J is the original loss function, represents partial derivative;
[0022]
[0023] The formula for adaptive weight decay can be expressed as:
[0024] γ i =γ base ·exp(-θ·H ii )
[0025] In the formula, θ is the coefficient that controls the attenuation speed. Generally, it is necessary to find the optimal coefficient through multiple adjustments during use;
[0026] When using adaptive weight decay adjustment, the γ of the original formula is base Replace with γ i , the final loss function is:
[0027]
[0028] Preferably, in step S5, for a set of regularization parameter candidate values γ={γ1,γ2,...γ n}, using grid search, select the regularization parameter that can achieve the best performance on the validation set, expressed as:
[0029]
[0030] Where, L total (γ) is the loss function with regularization parameter γ calculated on the validation set.
[0031] The substantial effect brought about by the present invention is that by introducing the L2 regularization term, the loss function of the model is modified, and larger weight values in the model are penalized, thereby effectively reducing the complexity of the model, improving the generalization ability of the model, and reducing the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a flow chart of a loss function optimization method of the present invention. DETAILED DESCRIPTION
[0033] The technical solution of the present invention is further described in detail below through embodiments and in conjunction with the accompanying drawings. Embodiment: When L2 regularization is used to penalize the weights of a model, a regularization term is usually added to the loss function. For logistic regression, L2 regularization is to add a term to the loss function:
[0034]
[0035] Where f(x,a) is a loss function used to measure the difference between the model's predicted value and the true value, and a regularization term is added. γ is the regularization parameter, is the L2 norm of the vector parameter w. γ is a hyperparameter that needs to be adjusted. m represents the number of samples in the model training data, that is, the number of samples in the dataset. J loss function is used to calculate the predicted value of the i-th sample and the true value y i The difference between. Normalization factor, used to average the total loss over each sample. For general neural networks, regularization is defined as:
[0036]
[0037] in Sum the squares of all elements:
[0038]
[0039] If the regularization parameter γ is set large enough, the weight matrix W will be set to a value close to 0, the neural network will be simplified, and overfitting will be less likely to occur.
[0040] Specifically, it can be a cross entropy loss function (for classification tasks) or a mean square error loss function (for regression tasks), and the regularization term penalizes the larger weight values in the model. In this way, by adjusting the size of the regularization parameter γ, the influence of the regularization term in the loss function can be controlled, thereby achieving the purpose of reducing overfitting.
[0041] During the training process, the back propagation algorithm takes into account the impact of the regularization term on the loss function, thereby updating the model parameters to minimize the overall loss function J. In this way, the model will not only minimize the loss of training data during the learning process, but also try to keep the weight parameters moderate to avoid overfitting.
[0042] A loss function optimization method of an image detection model in this embodiment is as follows: Figure 1 As shown, the loss function during model training is corrected in the following way:
[0043] S1. Add a regularization term to the loss function and initialize the weight decay parameter of each weight parameter;
[0044] S2. After each iteration or a certain number of iterations, dynamically adjust the weight decay parameter corresponding to the weight parameter according to the importance of the weight parameter and the level at which it is located;
[0045] S3, applying the dynamically adjusted weight decay parameter to the loss function and calculating the regularization term;
[0046] S4, looping steps S2 and S3, continuously optimizing the regularization parameter during the entire training process, and obtaining a set of regularization parameter candidate values; machine learning will obtain a set of parameters with different effects, some of which may have better recall, some may have better accuracy, and some may have better AP;
[0047] S5. Evaluate the performance of the model on the validation set and select the best regularization parameter configuration.
[0048] In step S1, the loss function after adding the regularization term is:
[0049]
[0050] m is the number of samples in the model training data, that is, the number of samples in the data set; y i is the true value of the i-th sample, is the predicted value of the i-th sample, L is the total number of network layers of the image detection model, N l is the total number of weight parameters of the lth layer, w li is the i-th weight parameter of the l-th layer, γ li is the i-th weight parameter w of the l-th layer li The weight decay parameter of the L2 regularization method. The traditional L2 regularization method usually applies the same regularization treatment to the weight parameters of all levels, while in this scheme, the importance of local feature information in the regularization process is considered. Specifically, for the weight parameters of the lth layer, different regularization strategies can be adopted according to the level and the corresponding feature information. For example, the local feature information can be integrated into the weight decay parameter to combine it with the importance of the weight parameter. When combining the local feature information, different regularization treatments can be applied to the weight parameters of different levels by adjusting the weight decay parameter.
[0051] In step S2, the weight decay parameter is specifically:
[0052] γ li =α·γ base +β·g li
[0053] In the formula, γ li is the i-th weight parameter w of the l-th layer li The weight decay parameter, α and β are adjustment coefficients, γbase is the basic regularization parameter, g li is the weight parameter w li The evaluation value of γ base Generally, empirical settings are used. g is used as the initial evaluation value, and an initial importance evaluation value is set for the weight parameter based on domain knowledge or previous research experience.
[0054] α and β train the model on the training set, then evaluate the impact of different regularization parameter values on model performance on the validation set, and finally select the regularization parameter that achieves the best performance.
[0055] The weight attenuation parameters corresponding to the weight parameters are dynamically adjusted according to the importance of the weight parameters and the level they are at:
[0056] The Hessian matrix is used to evaluate the importance of each weight parameter:
[0057]
[0058] In the formula, H ii is the element in the ith row and ith column of the Hessian matrix, and J is the original loss function, which is a measure of the difference between the model prediction and the actual value. The goal of neural network training is to minimize the loss function. Represents the partial derivative, that is, the weight parameter w i The diagonal elements of the Hessian matrix; The importance of weight parameters is evaluated here using the Hessian matrix, which is the second-order derivative matrix of the loss function, which contains the interaction information between weight parameters.
[0059]
[0060] The formula for adaptive weight decay can be expressed as:
[0061] γ i =γ base ·exp(-θ·H ii )
[0062] In the formula, θ is the coefficient that controls the attenuation speed. Generally, it is necessary to find the optimal coefficient through multiple adjustments during use;
[0063] When using adaptive weight decay adjustment, the γ of the original formula is base Replace with γ i , the final loss function is:
[0064]
[0065] In the traditional L2 regularization method, the regularization parameter γ often needs to be set manually, and its value is usually fixed. In this scheme, an adjustable regularization parameter setting is proposed. Specifically, the value of the regularization parameter can be automatically adjusted according to the characteristics of the actual data set and the complexity of the model structure. For example, the optimal regularization parameter can be determined by cross-validation or other adaptive adjustment methods, so that it can better adapt to different application scenarios and data set characteristics.
[0066] In terms of adjustable regularization parameters, the best regularization parameter can be determined by cross-validation or other adaptive adjustment methods. A common method is to use grid search to search within a specified parameter range and select the regularization parameter value that can achieve the best performance on the validation set. That is, in step S5, for a set of regularization parameter candidate values γ={γ1,γ2,...γ n}, using grid search, select the regularization parameter that can achieve the best performance on the validation set, expressed as:
[0067]
[0068] Where, L total (γ) is the loss function with regularization parameter γ calculated on the validation set.
[0069] This solution can handle the regularization problem of the model more flexibly, reduce the overfitting problem of the neural network model, and thus improve the generalization ability and performance of the model.
[0070] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
[0071] Although the terminology such as regularization and weight decay parameter is used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.
Claims
1. A loss function optimization method for an image detection model, characterized in that: The loss function during model training is modified in the following way: S1. Add a regularization term to the loss function and initialize the weight decay parameter of each weight parameter; S2. After each iteration or a certain number of iterations, dynamically adjust the weight decay parameter corresponding to the weight parameter according to the importance of the weight parameter and the level at which it is located; S3, applying the dynamically adjusted weight decay parameter to the loss function and calculating the regularization term; S4, looping steps S2 and S3, continuously optimizing the regularization parameter during the entire training process, and obtaining a set of candidate values of the regularization parameter; S5. Evaluate the performance of the model on the validation set and select the best regularization parameter configuration from the regularization parameter candidate values.
2. The method for optimizing the loss function of an image detection model according to claim 1, characterized in that: In step S1, the loss function after adding the regularization term is: m is the number of samples in the model training data, that is, the number of samples in the data set; y i is the true value of the i-th sample, is the predicted value of the i-th sample, L is the total number of network layers of the image detection model, N l is the total number of weight parameters of the lth layer, w li is the i-th weight parameter of the l-th layer, γ li is the i-th weight parameter w of the l-th layer li The weight decay parameter of .
3. The method for optimizing the loss function of an image detection model according to claim 1 or 2, characterized in that: In step S2, the weight decay parameter is specifically: c li =a·c base +β·g li In the formula, γ li is the i-th weight parameter w of the l-th layer li The weight decay parameter, α and β are adjustment coefficients, γ base is the basic regularization parameter, g li is the weight parameter w li The assessed value.
4. The method for optimizing the loss function of an image detection model according to claim 1 or 2, characterized in that: The weight attenuation parameters corresponding to the weight parameters are dynamically adjusted according to the importance of the weight parameters and the level they are at: The Hessian matrix is used to evaluate the importance of each weight parameter: In the formula, H ii is the element in the i-th row and i-th column of the Hessian matrix, J is the original loss function, represents partial derivative; The formula for adaptive weight decay is expressed as: c i =c base ·exp(-θ·H ii ) In the formula, θ is the coefficient that controls the decay speed; When using adaptive weight decay adjustment, the γ of the original formula is base Replace with γ i , the final loss function is:
5. The method for optimizing the loss function of an image detection model according to claim 3, characterized in that: In step S5, for a set of regularization parameter candidate values γ={γ1,γ2,...γ n }, using grid search, select the regularization parameter that can achieve the best performance on the validation set, expressed as: Where, L total (γ) is the loss function with regularization parameter γ calculated on the validation set.