A re-weighted long-tailed distribution image recognition method based on sample forgetting

Abstract

The application discloses a kind of based on sample forgetful reweighting long tail distribution image recognition method, comprising: S1, pre-training obtains sample forgetful information;S2, the total amount of unsmoothed class effective information is calculated;S3, the total amount of class effective information obtained is smoothed;S4, design weight factor, introduce weight factor into loss function, obtain the reweighting loss function based on sample forgetfulness;S5, select part of data in training set and input into defined neural network model, obtain the output of softmax layer by neural network calculation;S6, according to the reweighting loss function, the reweighting loss of each batch is calculated;S7, using optimization algorithm and according to the expression of loss function and chain rule, the gradient and network parameter are updated;S8, repeat steps S5-S7 until total loss function converges, input test set sample data, and the recognition result is calculated.The application adjusts the proportion of each class loss in total loss, and alleviates the imbalance of gradient proportion caused by long tail distribution.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, specifically relating to a reweighted long-tailed distribution image recognition method based on sample forgetting. Background Technology

[0002] In the real world, the sample sizes of different classes are often imbalanced, and the datasets exhibit a long-tailed distribution. There are various optimization methods to address recognition problems under this long-tailed distribution, the simplest being resampling of the data and reweighting the loss. Reweighting adjusts the proportion of each class's loss in the total loss, mitigating the gradient imbalance caused by the long-tailed distribution. Unlike other methods, reweighting offers flexibility and convenience in loss function calculation, making it a preferred approach for many complex tasks such as object detection and instance segmentation to address long-tailed distribution problems.

[0003] The key to reweighting methods lies in designing the weighting mechanism. The most intuitive approach is to directly reweight the loss using the label frequencies of the training samples. Yin Cui et al., authors of the class-balanced loss function, proposed a new theoretical framework to characterize data overlap: samples whose information does not overlap with other samples contribute 1 to the total information of their class, while samples with overlapping information contribute less than 1. The total information contained in a class when overlap is possible is called the effective number of samples for that class. Yin Cui added a class-balanced reweighting term, inversely proportional to the effective number of samples, to the loss function to construct the class-balanced loss function. However, the class-balanced loss function has two drawbacks: first, all classes share the same hyperparameter to fit the effective number of samples for each class, whereas in reality, each class should have its own effective number of samples, independent of other classes; second, the parameter tuning time increases significantly, as the class-balanced loss function requires hyperparameter adjustment without a fixed optimal hyperparameter value reference, resulting in high training costs.

[0004] In model training, a large amount of training data contains some difficult-to-distinguish negative samples. Identifying these negative samples and then training them specifically can improve the model's accuracy. In 2018, Toneva et al. studied the importance of data from the perspective of "forgetting," i.e., misclassification. Toneva et al. found that samples that are rarely forgotten (misclassified) have little impact on the final test accuracy; on the contrary, samples that are easily forgotten will affect the final evaluation results. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a reweighted long-tailed distribution image recognition method based on sample forgetting.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A reweighted long-tailed distribution image recognition method based on sample forgetting includes the following steps:

[0008] S1. Pre-train the training set and record the number of times each sample is forgotten and the maximum number of times all samples are forgotten during pre-training.

[0009] S2. Calculate the total amount of unsmoothed effective class information N based on the number of forgetting events for each sample. i ;

[0010] S3, the total amount of valid category information obtained N i Smoothing Based on the total amount of effective information of the smoothed categories Calculate a category containing n i The number of valid samples when there are 1 sample

[0011] S4, Valid Sample Count of Design and Class Inversely proportional weighting factor α i By incorporating weighting factors into the loss function, we obtain the reweighted loss function L based on sample forgetting. * ;

[0012] S5. Select a portion of the data from the training set and input it into the predefined ResNet18 neural network model. Calculate the output of the softmax layer using ResNet18.

[0013] S6. Based on the reweighted loss function L obtained in step S4, which is based on sample forgetting. * The formula calculates the reweighted loss for each batch. * ;

[0014] S7. Use an optimization algorithm based on stochastic gradient descent and calculate the gradient Φ and update the Resnet18 network parameters according to the expression of the loss function and the chain rule.

[0015] S8. Repeat steps S5-S7 until the total loss function L is reached. * Convergence: Define the ResNet18 network using converged model parameters, input test set sample data, and calculate the recognition results.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0017] 1. The method of this invention assigns low loss weights to categories with high effective sample sizes and high loss weights to categories with low effective sample sizes; it uses category forgetting information to fit the effective sample size of each category, thus determining the effective sample size of each category and making the parameters in the weighting factors more consistent. It can be calculated using formulas without manual parameter adjustment, thus greatly saving parameter adjustment time.

[0018] 2. This invention addresses the problems of existing methods in determining the effective number of samples for each category and the need for complex and time-consuming hyperparameter tuning. It not only retains the applicability of the method, making it applicable to most existing models and loss functions, but also improves the recognition accuracy based on efficient training, and performs better when the long-tail phenomenon is more prominent. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention;

[0020] Figure 2 This is a diagram illustrating the process of calculating the number of sample forgetting times in each minimum batch during the pre-training of this invention.

[0021] Figure 3 This is a schematic diagram of the composition structure of a single residual block;

[0022] Figure 4 This is a schematic diagram of the ResNet18 network structure. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0024] Example

[0025] like Figure 1 As shown, this invention provides a reweighted long-tailed distribution image recognition method based on sample forgetting, comprising the following steps:

[0026] S1. Pre-train the training set, recording the number of forgettings for each sample and the maximum number of forgettings for all samples during pre-training; assuming there are M samples in the dataset, use an array T_all = [T1, T2, T3, ..., T...]. M To record the number of times T is forgotten for each sample. i , i∈[1,M]; the maximum number of forgettings for all samples is denoted by T.

[0027] like Figure 2 As shown, T i The specific calculation steps for T are as follows:

[0028] Suppose there are M samples in the dataset, and define the number of forgetting times T for a given sample i. iAnd classification accuracy P i ; where, the initial time T i =0 and P i =0;

[0029] During training, if the recorded sample i is in the current training mini-batch, its forgetting count and classification accuracy will be updated.

[0030] After the current minimum batch training is completed, the classification result of the recorded sample i is obtained. If the classification is correct, the classification accuracy P = 1 of the samples recorded in the current minimum batch is recorded; otherwise, P = 0 is recorded.

[0031] If P P If T > P, it means that sample i was correctly classified in the previous recording but sample P was misclassified in the current recording. Therefore, update T. i =T i +1 and P i =P, otherwise do not update;

[0032] After all rounds of pre-training are completed, save the number of forgetting times T_all for all samples and find the maximum number of forgetting times T for all samples, i.e., T = max[T1, T2, T3, ..., T]. M ].

[0033] S2. Calculate the total amount of unsmoothed effective class information N based on the number of forgetting events for each sample. i ;

[0034] Total amount of effective information N for unsmoothed categories i The specific calculation method is as follows:

[0035]

[0036] Where T is the maximum number of forgettings across all samples. Let be the number of samples in the i-th class that were forgotten t times.

[0037] S3, the total amount of valid category information obtained N i Smoothing Based on the total amount of effective information of the smoothed categories Calculate a category containing n i The number of valid samples when there are 1 sample

[0038] The total amount of valid information N obtained for each category i Smoothing Specifically:

[0039]

[0040] Where, n i Let Avg be the number of samples in a given category, and let Avg be the mean of all samples across all categories. C represents the total number of categories in the dataset;

[0041] A certain category contains n i The number of valid samples when there are 1 sample Specifically:

[0042]

[0043] in,

[0044] S4, Valid Sample Count of Design and Class Inversely proportional weighting factor α i By incorporating weighting factors into the loss function, we obtain the reweighted loss function L based on sample forgetting. * ;

[0045] The weighting factor α is inversely proportional to the number of effective samples in the class. i The specific design is as follows:

[0046]

[0047] Reweighted loss function L based on sample forgetting * Specifically:

[0048]

[0049] Where L(p,y) is a regular non-reweighted loss function, p represents the network output after processing by the sigmoid function or softmax function, and y represents the sample label.

[0050] S5. Select a portion of the data from the training set and input it into the defined neural network model; this embodiment uses ResNet18 as the neural network example. ResNet18 is a classic residual network; the "18" refers to 18 layers with weights, including convolutional layers and fully connected layers, but excluding pooling layers and normalization layers. The ResNet18 network contains four residual blocks, each containing two 3×3 convolutional layers with the same number of output channels. Each convolutional layer is followed by a normalization layer and a ReLU activation function. Then, through the cross-layer data path, these two convolutional operations are skipped, and the input is directly added before the final ReLU activation function. For example... Figure 3 The diagram shown is a schematic representation of the composition of a residual block. Figure 4 The image shows a schematic diagram of the ResNet18 network model.

[0051] Next, the output of the softmax layer is calculated using the ResNet18 network. The specific expression for the softmax function is as follows:

[0052]

[0053] Among them, z i The output of the network is given by p, where e is the natural constant, C is the total number of classes in the dataset, and p is the total number of classes in the dataset. i This is the network output after processing by the softmax function.

[0054] S6. Based on the reweighted loss function L obtained in step S4, which is based on sample forgetting. * The formula calculates the reweighted loss for each batch. * .

[0055] S7. Use an optimization algorithm based on stochastic gradient descent to update the Resnet18 network parameters;

[0056] Because batch gradient descent calculates the partial derivative of the entire dataset with respect to each parameter every time it updates, the training process becomes extremely slow as the number of samples increases. Stochastic gradient descent, on the other hand, updates the dataset iteratively by calculating the partial derivative of a single batch of samples with respect to the corresponding parameter.

[0057] The gradient Φ is calculated using the expression for the loss function and the chain rule as follows:

[0058]

[0059] Where Θ represents the neural network parameters, and Φ is the gradient of the corresponding parameters with respect to a single batch of data.

[0060] The specific steps for updating ResNet18 network parameters are as follows:

[0061] Θ new =Θ-ηΦ

[0062] Where η represents the pre-set learning rate parameter, and Θ new This represents new parameters learned from batch data.

[0063] S8. Repeat steps S5-S7 until the total loss function L is reached. * Convergence: Define the network using converged model parameters, input test set sample data, and calculate the recognition result.

[0064] It should also be noted that, in this specification, terms such as "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for reweighted long-tailed distribution image recognition based on sample forgetting, characterized in that, Includes the following steps: S1. Pre-train the training set and record the number of times each sample is forgotten and the maximum number of times all samples are forgotten during pre-training. S2. Calculate the total amount of unsmoothed effective class information N based on the number of forgetting events for each sample. i ; S3, the total amount of valid category information obtained N i Smoothing Based on the total amount of effective information of the smoothed categories Calculate a category containing n i The number of valid samples when there are 1 sample S4, Valid Sample Count of Design and Class Inversely proportional weighting factor α i By introducing the weighting factor into the loss function, we obtain the reweighted loss function L* based on sample forgetting; S5. Select a portion of the data from the training set and input it into the predefined ResNet18 neural network model. Calculate the output of the softmax layer using ResNet18. S6. Calculate the reweighted loss l* for each batch according to the formula of the reweighted loss function L* based on sample forgetting obtained in step S4. S7. Use an optimization algorithm based on stochastic gradient descent and calculate the gradient Φ and update the Resnet18 network parameters according to the expression of the loss function and the chain rule. S8. Repeat steps S5-S7 until the reweighted loss function L* converges. Define the ResNet18 network using the converged model parameters, input the test set sample data, and calculate the recognition result.

2. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 1, characterized in that, Step S1 is as follows: Suppose there are M samples in the dataset, and we use an array T_all = [T1, T2, T3, ..., T... M To record the number of times T is forgotten for each sample. i , i∈[1,M]; the maximum number of forgettings for all samples is denoted by T.

3. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 2, characterized in that, T i The specific calculation steps for T are as follows: Suppose there are M samples in the dataset, and define the number of forgetting times T for a given sample i. i And classification accuracy P i ; where, the initial time T i =0 and P i =0; During training, if the recorded sample i is in the current minimum batch of training, its forgetting count and classification accuracy will be updated. After the current minimum batch training is completed, the classification result of the recorded sample i is obtained. If the classification is correct, the classification accuracy P = 1 of the samples recorded in the current minimum batch is recorded; otherwise, P = 0 is recorded. If P i >P indicates that sample i was correctly classified in the previous recording but was misclassified in the current recording. Therefore, update T. i =T i +1 and P i =P, otherwise do not update; After all rounds of pre-training are completed, save the number of forgetting times T_all for all samples and find the maximum number of forgetting times T for all samples, i.e., T = max[T1, T2, T3, ..., T]. M ].

4. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 2, characterized in that, In step S2, the total amount of unsmoothed category effective information N i The specific calculation method is as follows: Where T is the maximum number of forgettings across all samples. Let be the number of samples in the i-th class that were forgotten t times.

5. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 4, characterized in that, In step S3, the total amount of valid category information N obtained is... i Smoothing Specifically: Where, n i Let Avg be the number of samples in a given category, and let Avg be the mean of all samples across all categories. C represents the total number of categories in the dataset; A certain category contains n i The number of valid samples when there are 1 sample Specifically: in, 6. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 5, characterized in that, In step S4, the weighting factor α is inversely proportional to the number of valid samples in the class. i The specific design is as follows: The reweighted loss function L* based on sample forgetting is as follows: Where L(p,y) is a regular non-reweighted loss function, p represents the network output after processing by the sigmoid function or softmax function, and y represents the sample label.

7. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 1, characterized in that, In step S5, ResNet18 is a residual network. The 18 refers to 18 layers with weights, including convolutional layers and fully connected layers, but excluding pooling layers and normalization layers. The ResNet18 network contains 4 residual block groups. Each residual block includes two 3×3 convolutional layers with the same number of output channels. Each convolutional layer is followed by a normalization layer and a ReLU activation function. Then, through the cross-layer data path, these two convolutional operations are skipped, and the input is directly added before the final ReLU activation function.

8. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 5, characterized in that, In step S5, when the output of the softmax layer is calculated using the ResNet18 network, the specific expression for the softmax function is as follows: Among them, z i The output of the network is given by p, where e is the natural constant, C is the total number of classes in the dataset, and p is the total number of classes in the dataset. i This is the network output after processing by the softmax function.

9. The image recognition method based on sample forgetting and reweighted long-tail distribution according to claim 6, characterized in that, In step S7, the gradient Φ is calculated based on the expression of the loss function and the chain rule as follows: Where Θ represents the Resnet18 network parameters, and Φ is the gradient of the corresponding parameters with respect to a single batch of data; The ResNet18 network parameters are updated using an optimization algorithm based on stochastic gradient descent. The specific steps for updating the ResNet18 network parameters are as follows: I new =Θ-ηΦ Where η represents the pre-set learning rate parameter, and Θ new This represents new parameters learned from batch data.

Images