A classification method and system for linear rectified gradient balance loss functions

Abstract

This invention proposes a novel classification method using a linear rectified gradient balancing loss function. The method includes the following steps: defining a linear rectified gradient balancing loss function; constructing a model and initializing its parameters; acquiring and preprocessing the dataset; and iteratively training the model according to a preset method. The test accuracy of the model trained by this invention is significantly higher than that of the other four comparative models, even exceeding the test accuracy of the comparative models after 200 iterations. The model's classification accuracy is improved, and the gradient updates are more effective in the early stages of training, accelerating model convergence. Upon completion of training, the model trained by this invention has the highest test accuracy and the highest F1 score for minor class samples compared to the control model, indicating that it can most effectively learn the knowledge of minor class samples in the dataset. The model exhibits higher accuracy and recall for minor class samples.

Description

Technical Field

[0001] This invention belongs to the technical field of data processing, specifically relating to a novel classification method and system for linear rectified gradient balance loss functions. Background Technology

[0002] The unequal number of samples in each category of a dataset can lead to training problems for many machine learning classification models, including deep learning classification models. This phenomenon is known as the data imbalance problem. Real-world data is often imbalanced. For example, in rare disease diagnosis, the ratio of patients to those without the disease is vastly different. Similarly, in street scene recognition tasks, elements such as signals and bicycles occupy a very small percentage of the image pixels. These smaller sample values ​​often contain crucial information for the corresponding learning task. Therefore, addressing the data imbalance problem can help broaden the application areas of deep learning classification models.

[0003] More specifically, most text classification problems (such as spam classification, user review classification, etc.) are inherently imbalanced data problems. When using traditional cross-entropy as the loss function, it may lead to reduced model accuracy, reduced recall of minor class samples, and training difficulties.

[0004] In view of this, it is very meaningful to propose a new classification method and system for linear rectified gradient balance loss function. Summary of the Invention

[0005] To address the problems of reduced model accuracy, decreased recall of minor class samples, and training difficulties in existing models when dealing with imbalanced data, this invention provides a novel classification method and system based on the linear rectified gradient balancing loss function, thereby resolving the existing technical deficiencies.

[0006] Firstly, this invention proposes a novel classification method for linear rectified gradient balancing loss functions, which includes the following steps:

[0007] Define a linear rectified gradient balancing loss function;

[0008] Build the model and initialize its parameters;

[0009] Obtain the dataset and preprocess it;

[0010] The model is iteratively trained according to a preset method.

[0011] Preferably, the function includes a linear rectification loss function L. ic The definition is as follows:

[0012] L ic (x)=x·(β(x)-z)

[0013]

[0014] Where x is the classification prediction value of the sample output through the forward propagation of the network, z is the sample class label, and β is the Sign function;

[0015] The function also includes a gradient balance loss function L. c When the network uses Sigmoid as the output layer activation function, it is defined as follows:

[0016]

[0017] Where x is the classification prediction value of the sample output by the forward propagation of the network, z is the sample class label, β is the Sign function, σ is the Sigmoid function, w0 and w1 are the sum of gradients calculated for samples with label 0 or label 1 respectively, C1 is the set of logits of the correctly classified positive samples in the current training batch, C0 is the set of logits of the correctly classified negative samples in the current training batch, and w0 and w1 do not participate in backpropagation.

[0018] Preferably, the constructed model is a two-layer LSTM model, wherein each layer contains 64 hidden units.

[0019] Preferably, the dataset is a Yelp dataset, and the preprocessing of the dataset includes:

[0020] 150,000 data points with text lengths ranging from 10 to 300 words were selected as the training dataset, and 10,000 data points of the same length range were selected as the test dataset.

[0021] Each word is transformed into an 8000-dimensional vector using the word2vec algorithm.

[0022] Further preferably, preprocessing the dataset also includes:

[0023] User ratings with data labels of 1-2 are considered negative ratings;

[0024] Construct a text binary classification problem by using user ratings with data labels of 3-5 as positive evaluations;

[0025] The ratio of the number of samples in the major category to the number of samples in the minor category is 3.69.

[0026] More preferably, iterative training of the model according to a preset method specifically includes:

[0027] A batch of samples is input into the network, and the class prediction values ​​output by the forward propagation are calculated.

[0028] For misclassified samples, the loss value of the sample is calculated using the linear rectified loss function;

[0029] For correctly classified samples, the gradient balancing loss function is used to calculate the loss value of the sample;

[0030] Calculate the weighted sum of the gradients of the two loss functions, with the weighting coefficients being positive constants. In the experiment, the weighting coefficients are set to 1.

[0031] Update network parameters using the gradient of weighted summation;

[0032] The optimizer used to train the network is the Adam optimizer, which performs 1000 rounds of iterative optimization at a learning rate of 0.0001.

[0033] Preferably, the method further includes: using the same parameter settings, training an LSTM model with cross-entropy loss, weighted cross-entropy, focal loss and class balance loss respectively on the same dataset, and comparing it with a model trained with the linear rectified gradient balance loss function.

[0034] Secondly, this invention also proposes a novel classification system for linear rectified gradient balance loss functions, comprising:

[0035] Function definition module: Used to define the linear rectified gradient balancing loss function;

[0036] Model building module: Used to build multiple required models and initialize the model parameters;

[0037] Dataset Acquisition Module: Used to acquire the required dataset;

[0038] Dataset preprocessing module: Used to preprocess the acquired dataset;

[0039] Model Iterative Training Module: Used to iteratively train the model according to a preset method;

[0040] The comparison module is used to compare the results of training an LSTM model using cross-entropy loss, weighted cross-entropy, focal loss, and class balance loss with a model trained using the linear rectified gradient balance loss function.

[0041] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0042] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0043] Compared with the prior art, the beneficial results of the present invention are as follows:

[0044] (1) In the early stage of model training, the test accuracy of the model trained by this invention is much higher than that of the other four comparison models, and even higher than the test accuracy of the comparison models after 200 iterations. The model classification accuracy is improved, the gradient update is more effective in the early stage of model training, and the model convergence is accelerated. When the training is completed, the model trained by this invention has the highest test accuracy compared with the control model.

[0045] (2) The model trained by this invention is least affected by imbalanced datasets and can get rid of gradient traps and incorrect optimization problems caused by imbalanced data with the fewest iterations. The number of iterations consumed from the start of training to the start of learning small class sample information is only 17.3% of the traditional cross-entropy loss optimization method.

[0046] (3) When training is completed, the model trained by this invention has the highest F1 score for minor samples compared to the control model, that is, it is able to learn the knowledge of minor samples in the dataset most effectively, and the model has higher accuracy and recall for minor samples. Attached Figure Description

[0047] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0048] Figure 1 This is a comparison diagram of the training process of the neural network with the piecewise loss function of this invention and the traditional loss function;

[0049] Figure 2 This is a flowchart illustrating the new linear rectified gradient balance loss function classification method according to an embodiment of the present invention.

[0050] Figure 3 In the examples of this invention, (a) and (b) are graphs of the cross-entropy loss function and the ReL loss function used to calculate the loss value of the negative class sample in a binary classification problem.

[0051] Figure 4 This is a flowchart illustrating the new linear rectified gradient balance loss function classification system according to an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0053] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0054] This invention proposes a novel form of deep learning loss function and a corresponding model optimization method, which can help models avoid training difficulties and improve model performance when dealing with imbalanced data. The new loss function is called the Rectified Linear Gradient Balanced Loss (ReL-GBLoss). Unlike traditional loss functions, ReL-GB loss calculates the sample loss value in the form of a piecewise function, thereby avoiding the situation where the gradient provided by the large class samples in the imbalanced dataset drowns out the small class samples, thus helping the model to learn the small class data knowledge in the imbalanced dataset more effectively.

[0055] Most text classification problems (such as spam classification, user review classification, etc.) are inherently imbalanced data problems. When using traditional cross-entropy as the loss function, it may lead to reduced model accuracy, reduced recall of minor class samples, and training difficulties. We applied ReL-GB loss to an LSTM text binary classification model and compared it with cross-entropy loss, weighted cross-entropy, focal loss, and class-balanced loss. The new loss function has the following advantages: (1) the model classification accuracy is improved; (2) the model has higher accuracy and recall for minor class samples; (3) the gradient update is more effective in the early stage of model training, which can accelerate the convergence of the model.

[0056] This invention proposes a novel loss function to avoid the problem of imbalanced data, called Rectified Linear Gradient Balanced Loss (ReL-GB Loss). ReL-GB has two characteristics that help improve the model's test accuracy and recall rate of minor class samples on imbalanced datasets, and reduce the training time required: (1) a piecewise function, which calculates the gradient for correctly classified training samples and misclassified training samples respectively; and (2) dynamic weighting coefficients, which dynamically change the weighting coefficients of the loss function for each type of sample.

[0057] On the one hand, the introduction of piecewise functions can effectively shorten the time required for the model to reach a better state. This is because the process of model gradient optimization can be seen as the combined effect of the training gradients of each sample. The gradient of each training sample affects the update of network parameters. However, balancing the gradients of each sample can also cause the update trends of network parameters to cancel each other out, thus slowing down the network optimization process. Therefore, this section presents a two-stage piecewise loss function that only performs weighted balancing on a portion of the data. It can ensure that the model can quickly escape the gradient trap in the early stages of training, while also ensuring a reasonable and stable gradient update in the later stages of training.

[0058] On the other hand, the introduction of dynamic weighting coefficients can improve the final accuracy of model convergence. This is because using a constant-weighted loss function will change the data distribution fitted by the network, which may impair the classification accuracy of large-class samples, thus leading to a decrease in the overall classification performance of the network. Therefore, the weighting coefficients should be dynamically changing variables, just like the focal loss function, to avoid the data distribution misestimation problem caused by constant-weighted coefficients and ensure the accuracy performance of the model at convergence.

[0059] A comparison diagram of the neural network training process using traditional loss functions and piecewise loss functions is shown below. Figure 1 As shown, (a) is the network model training process using the traditional loss function; and (b) is the network model training process using the piecewise loss function.

[0060] The following section uses the LSTM model to handle the Yelp dataset classification task as an example to illustrate the technical solution of this invention in detail.

[0061] Figure 2 An embodiment of the present invention discloses a novel classification method for linear rectified gradient balance loss functions, such as... Figure 2 As shown, the method includes the following steps:

[0062] S1. Define the linear rectified gradient balance loss function;

[0063] Specifically, the function includes the linear rectification loss function L.ic The definition is as follows:

[0064] L ic (x)=x·(β(x)-z)

[0065]

[0066] Where x is the classification prediction value of the sample output through the forward propagation of the network, z is the sample class label, and β is the Sign function.

[0067] In imbalanced datasets, the number of samples in the larger class is often too high. Directly using the cross-entropy activation function can cause the gradients from the larger class to overwhelm the gradients from the smaller class. In extreme cases, this can lead to all samples being classified as belonging to the larger class. The new activation function is designed to only compute the gradients of samples misclassified by the model. This way, if all samples in the larger class are correctly classified, the gradient updates for the model parameters will be entirely determined by the gradients from the smaller class, allowing the model to quickly escape the aforementioned extreme situation.

[0068] The definition of the cross-entropy loss function indicates that the larger the predicted label value of a misclassified sample, the larger the gradient provided for the update. That is, classification errors have different degrees; samples with greater error severity provide larger gradients, which helps the model converge quickly and is very effective for ideal datasets without contamination. In real-world datasets, a small amount of contaminated data is inevitable. This contaminated data may be inaccurately labeled samples or unrepresentative extreme cases within a certain class. The gradients provided by such contaminated data are detrimental to model performance improvement. Furthermore, this contaminated data is often difficult for the network to classify correctly; a greater degree of error generates a larger gradient, exacerbating the training problem. Since only the loss of misclassified samples is considered, as training progresses and the dataset is gradually fitted, the number of samples providing gradients decreases, and these contaminated data will dominate the gradients for parameter updates. Therefore, two approaches can effectively avoid the influence of contaminated data: one is to introduce correctly classified data into the calculation of the loss function, which will be discussed in the next section; the other is to ensure that all misclassified samples have the same gradient in the loss function, thereby suppressing the influence of extreme samples.

[0069] Based on the above discussion, for misclassified samples, we designed a loss function with the following characteristics: (1) the loss for correctly classified samples is 0; (2) the gradient for misclassified samples is constant. In order to make it have a more reasonable value, the gradients at both ends of the new loss function are designed to have the same limit as the gradients at both ends of the Sigmoid function combined with the cross-entropy loss function, that is, the gradient for misclassified samples is a constant 1.

[0070] The loss function for misclassified samples can be derived from the linear rectified loss function L. ic Define where β is the binarization function, x is the unactivated predicted value of the sample, and z is the label. Due to the loss function L... ic It is similar to the activation function of a Rectified Linear Unit (ReLU), hence it is called the Rectified Linear Loss (ReLLoss). Figure 3 Figures (a) and (b) show the function graphs of the cross-entropy loss function and the ReL loss function for calculating the loss value of the negative class sample in the binary classification problem, respectively. The unactivated prediction value represents the output value of the network's logits.

[0071] The function also includes a gradient balance loss function L. c When the network uses Sigmoid as the output layer activation function, it is defined as follows:

[0072]

[0073]

[0074] Where x is the classification prediction value of the sample output by the forward propagation of the network, z is the sample class label, β is the Sign function, σ is the Sigmoid function, w0 and w1 are the sum of gradients calculated for samples with label 0 or label 1 respectively, C1 is the set of logits of the correctly classified positive samples in the current training batch, C0 is the set of logits of the correctly classified negative samples in the current training batch, and w0 and w1 do not participate in backpropagation.

[0075] Specifically, the design of the loss function for misclassified samples ensures that the model can escape gradient traps more quickly on imbalanced datasets. As the number of correctly classified samples gradually increases and the number of misclassified samples decreases, the loss function for correctly classified samples gradually becomes dominant, and it will be the determining factor for the model's final convergence performance. If the loss for correctly classified samples is missing in gradient updates, a few samples will determine the overall gradient in the later stages of training, potentially causing training instability and fluctuations in model accuracy. Subsequent experiments verified this hypothesis. When training the network using only the ReL loss function, after a certain number of iterations, when the number of misclassified samples decreases to a certain extent, the network will experience a significant drop in accuracy. At this point, the number of misclassified samples increases significantly, allowing the model to continue to optimize gradually. Then, after a certain amount of training, the accuracy drops again, and the cycle repeats. Therefore, calculating the loss for correctly classified samples is extremely important for the stability of model training.

[0076] The GB loss function, built upon the weighted cross-entropy loss function, is used to calculate the loss for correctly classified samples. Gradient weighting, like the weighting based on the inverse of the sample count, uses weighting to balance gradient imbalances caused by differences in the number of samples across classes. However, gradient weighting achieves a more accurate balance of gradients for updating various parameters due to the chain rule, where the gradient for updating all parameters in the network uses the derivative of the loss with respect to logits as a coefficient. Although, as discussed earlier, a more balanced gradient between the two classes implies slower parameter updates, the network training process is not significantly affected because the loss functions for correctly and incorrectly classified samples are calculated separately. In the early stages of training, the loss function for correctly classified samples dominates parameter updates, and the more balanced gradient ensures the stability of the model training. Furthermore, weighting and balancing only the loss for correctly classified samples effectively prevents the model from misestimating the dataset distribution, thus ensuring the final training accuracy of the model.

[0077]

[0078] The above formula defines the Gradient Balanced Loss (GB Loss) function, where x is the network's logits output, z is the sample's corresponding label, σ is the Sigmoid function, β is the binarization function, w0 is defined as the sum of gradients of the derivatives of the loss of correctly classified negative samples with respect to their logits, w1 is defined as the sum of gradients of the derivatives of the loss of correctly classified positive samples with respect to their logits, C is the logits value of all samples in the current training batch, C1 is the set of logits of correctly classified positive samples in the current training batch, and C0 is the set of logits of correctly classified negative samples in the current training batch. Furthermore, w0 and w1 do not participate in backpropagation.

[0079] S2. Build the model and initialize its parameters;

[0080] In this embodiment, the constructed model is a two-layer LSTM model, wherein each layer contains 64 hidden units.

[0081] S3. Obtain the dataset and preprocess it;

[0082] In this embodiment, the dataset is the Yelp dataset; in other embodiments, other datasets may also be used.

[0083] Specifically, preprocessing the dataset includes:

[0084] S31. Select 150,000 data entries with text lengths ranging from 10 to 300 words as the training dataset, and 10,000 data entries with the same length range as the test dataset.

[0085] S32. Convert each word into an 8000-dimensional vector using the word2vec algorithm.

[0086] Furthermore, preprocessing the dataset also includes:

[0087] S33. User ratings with data labels of 1-2 are considered negative ratings;

[0088] S34. Construct a text binary classification problem by using user ratings with data labels of 3-5 as positive evaluations;

[0089] The ratio of the number of samples in the major category to the number of samples in the minor category is 3.69.

[0090] S4. Iteratively train the model according to a preset method;

[0091] S41. Input a batch of samples into the network and calculate the category prediction value of the forward propagation output;

[0092] S42. For misclassified samples, the loss value of the sample is calculated using the linear rectified loss function;

[0093] S43. For correctly classified samples, the gradient balance loss function is used to calculate the loss value of the samples;

[0094] S44. Calculate the weighted sum of the gradients of the two loss functions. The weighting coefficients are positive constants. In the experiment, the weighting coefficients are set to 1.

[0095] S45. Update network parameters using weighted summation gradients;

[0096] The optimizer used to train the network is the Adam optimizer, which performs 1000 rounds of iterative optimization at a learning rate of 0.0001.

[0097] The function L for calculating the loss of correctly classified samples mentioned above. c The function L for calculating the loss of misclassified samples ic Neither of these two loss functions alone can guarantee successful training in a classification model. Their functions are complementary; only by combining them can excellent performance be achieved. They are combined into a single loss function through a weighted sum, which is the ReL-GB loss. The definition is as follows:

[0098] L(x)=L ic (x)+λ·L c (x)

[0099] Here, λ is a hyperparameter that takes a positive value and is used to adjust the proportion of gradient updates in the two loss functions. Theoretically, a larger value of λ makes the model training process smoother, while a smaller value makes the training process faster in the early stages. For simplicity, the value of λ is set to 1 in the experiments of this embodiment.

[0100] S5. Using the same parameter settings and the same dataset, train the LSTM model using cross-entropy loss, weighted cross-entropy, focal loss, and class balance loss respectively, and compare it with the model trained by the linear rectified gradient balance loss function.

[0101] Below, we will demonstrate a comparison of the Rectified Gradient Balanced Loss Function (ReL-GB) with the traditional Cross-Entropy Loss Function (CE), Focus Loss Function (FL), Class Balanced Loss Function (CBL) weighted by effective sample count, and Inverted Sample Frequency Cross-Entropy (ICF-CE) on imbalanced datasets through specific experiments. FL and CBL have been verified as two of the most effective methods among several common approaches to addressing data imbalance. They were applied to four different types of deep learning models and tested on three different types of datasets.

[0102] Four commonly used datasets were used in the experiments: the MNIST dataset, the CIFAR-10 and CIFAR-100 datasets, and the Yelp dataset. The MNIST dataset is a handwritten Arabic numeral dataset containing digits 0 to 9 in ten categories. The training set has 60,000 samples, and the test set contains 10,000 samples. Each sample is stored as a 784-dimensional vector, and the values ​​in each dimension have been normalized. It is one of the most commonly used datasets for feature extraction and classification tasks. In the experimental tasks, to verify the ability of various loss functions to handle imbalanced data, the MNIST dataset was divided into ten imbalanced data classification tasks based on the ten class labels. Each task corresponds to a binary classification task involving one set of digits and the other nine sets of digits. These are referred to as the imbalanced MNIST classification tasks in subsequent experiments. The experiments were conducted on each sub-task, and the performance metrics were averaged.

[0103] The CIFAR-10 dataset is a color image dataset containing three RGB channels. It includes ten categories, including several animals and vehicles. Each image is 32 pixels wide and long, and contains 50,000 training images and 10,000 test images. Similarly, the ten categories of CIFAR-10 were split into 10 binary classification sub-tasks in the experiments to validate the performance of various loss functions on the imbalanced dataset; this is referred to as the imbalanced CIFAR-10 classification task in subsequent experiments. The CIFAR-100 dataset is similar to the CIFAR-10 dataset, except that it contains 100 categories, which are further divided into 20 superclasses. In the experiments, independent binary classification sub-tasks were constructed based on these superclasses as the imbalanced CIFAR-100 classification task.

[0104] The Yelp dataset is a commonly used text classification dataset. Each sample contains a user text comment and the corresponding star rating given by the user. The star rating is divided into 5 levels. In the actual use of the experiment, the rating of 3 stars or more is regarded as the user's positive evaluation, while the rating of 2 stars or less is regarded as the user's negative evaluation. It is constructed as a binary classification problem. The comments with more than 300 words and less than 10 words are filtered out. The words are converted into 8000-dimensional vectors by the word2vec algorithm

[72] . 150,000 filtered data are selected as the training set and 10,000 data are selected as the test set. The ratio of the two classes of samples is 3.69.

[0105] The experiments used four different deep neural network architectures to verify the generality of the proposed loss function: FCN, CNN, ResNet, and LSTM. FCN was applied to the imbalanced MNIST feature classification task, CNN to the imbalanced CIFAR-10 image classification task, ResNet to the imbalanced CIFAR-100 image classification task, and LSTM to the Yelp dataset text classification task.

[0106] The FCN consists of three fully connected layers, each with dimensions of 86, 10, and 1 respectively. The CNN uses a VGG-16 architecture, comprising 13 convolutional layers and 3 fully connected layers. The ResNet network uses a ResNet-18 architecture, containing 8 residual modules. Each of the four phases of the network contains 2 residual modules, for a total of 18 computational layers. To adapt to the input data, the kernel size of the first convolutional layer is reduced from the original 7×7 size to 3×3 to improve model performance. The LSTM network consists of two stacked 64-dimensional network layers. All models are randomly initialized in the same way, using the Adam optimizer. The FCN, CNN, and LSTM use a learning rate of 1e-4, while ResNet uses a learning rate of 1e-5. Training is performed for 1000 epochs, and the highest validation accuracy and other evaluation metrics are recorded at epochs of 100, 200, 500, and 1000. In the experiments, the hyperparameter γ of FL was uniformly set to 2, while the hyperparameter β of CBL was set to 0.9999. These hyperparameter settings were all selected based on the guidelines of the original work. To ensure experimental fairness, except for the special form of FL, the weighting coefficients of the other loss functions were scaled proportionally to ensure that the sum of the coefficients was 2, thus avoiding differences in learning rates caused by weighting.

[0107] The experiment intentionally chose a relatively simple model structure as the basis for two reasons: first, complex models would reduce the performance gap between different loss functions, which would be detrimental to experimental comparison; second, in later chapters, it is necessary to apply the loss function to simpler models, so there are higher requirements for the performance of the loss function on smaller models.

[0108] The evaluation metrics for the experiment included classification accuracy, minor class recall, F1 score, and the number of training epochs required to escape gradient traps in the early stages of training. Classification accuracy is the most commonly used performance metric for classification models, reflecting the model's classification ability. Since the common use of weighted cross-entropy often leads to a decrease in the final accuracy of the model, this evaluation metric is even more significant for measuring the performance of imbalanced data classification algorithms. Minor class recall refers to the proportion of correctly classified samples in a smaller class out of the samples in that class. Minor class recall reflects the model's ability to resist erroneous gradients in imbalanced data and extract features from minor classes.

[0109] Since classes with fewer samples often hold greater importance in datasets, in applications focused on discovery and early warning, such as disease diagnosis, cybersecurity, and spam filtering, the recall rate of minority classes is more significant than the overall classification accuracy. Experiments show that the recall rate of minority classes corresponds to the highest model accuracy. The F1 score measures the average performance between model recall and accuracy. For imbalanced data, recall or accuracy alone is insufficient to measure a model's ability; therefore, the F1 score is a crucial indicator of a model's classification capability. We define the number of training epochs required to escape the gradient trap as the number of epochs from network initialization where the recall rate of minority classes first increases. For the aforementioned imbalanced data classification tasks, most models enter an erroneous convergence state in the early stages of training, classifying all samples into the larger class, and the recall rate of minority classes is reset to zero. This aligns with the formal derivation in this chapter. The shorter the duration of this state, the stronger the model's ability to escape the gradient trap. This ability directly determines the model's training efficiency on imbalanced datasets, and in extreme cases, even determines whether the model can train normally.

[0110] (1) This experiment demonstrates the performance of the FCN model on the imbalanced MNIST classification task by applying the ReL-GB loss function, CE loss function, ICF-CE loss function, FL function and CBL function respectively.

[0111] Table 1 shows the performance of models using different loss functions on the imbalanced MNIST classification task.

[0112]

[0113] As shown in Table 1, the network model applying the ReL-GB loss function achieved excellent accuracy after 100 iterations, making it the best among several loss functions. It achieved 6.96% higher accuracy than the model using CE, and even surpassed the accuracy of the other four functions after 200 iterations. This indicates that the model using the ReL-GB loss function can obtain effective gradient updates and quickly optimize in the early stages of training. Furthermore, the accuracy advantage of the model using the ReL-GB loss function was maintained at 200, 500, and 1000 iterations.

[0114] Regarding recall, the ICF-CE loss function achieved the highest recall on this dataset. However, its precision performance from 500 to 1000 iterations was the lowest among the four functions. This indicates that the model applying ICF-CE overestimated the loss for minority class samples, sacrificing model precision for better recall. The same issue exists with CBL; a higher weighting for minority class samples improves recall but reduces model precision. By combining the F1 scores of each experiment, ReL-GB shows the highest F1 score, indicating that ReL-GB has the strongest classification ability for minority class samples compared to other loss functions.

[0115] Regarding the speed of escaping the gradient trap, the CE loss function model took an average of 257.3 iterations to escape the gradient trap and begin normal optimization. This indicates that traditional classification models based on cross-entropy are severely affected by data imbalance. In contrast, ReL-GB only required 3.3 iterations to improve the recall of minority class samples. This means that ReL-GB can quickly and effectively learn information about minority class samples, requiring only 1.3% of the number of iterations required by CE, and is roughly on par with ICF-CE, which overestimates the loss of minority class samples. This demonstrates that ReL-GB can obtain more reasonable gradients from imbalanced datasets to train the model.

[0116] (2) This experiment demonstrates the performance of CNN models on the imbalanced CIFAR-10 image classification task by applying ReL-GB loss function, CE loss function, ICF-CE loss function, FL function and CBL function respectively.

[0117] Table 2 shows the performance of models using different loss functions on the imbalanced CIFAR-10 classification task.

[0118]

[0119] Similar to the previous experiment, the data in Table 2 shows that the model using the ReL-GB loss function also achieved the highest accuracy in 100, 200, 500, and 1000 iterations. This demonstrates that the ReL-GB loss function is effective for both convolutional networks and image classification tasks. The CBL model also performed comparably to ReL-GB at 1000 iterations. Combined with the previous experiment, this reflects to some extent that the performance of CBL depends on the appropriate setting of hyperparameters, while the ReL-GB function, using the same hyperparameters across different datasets, did not show significant fluctuations in model performance.

[0120] Regarding recall, ReL-GB achieved the highest recall rate for minor class samples and an F1 score tied with CBL in this task, indicating that ReL-GB can most effectively balance the update gradients of the two classes of samples in this task, thus ensuring both the accuracy and recall of the model.

[0121] Regarding the number of iterations required to escape the gradient trap, the ReL-GB model takes an average of 12.8 iterations to escape the gradient trap. Compared to the CE model, which represents most traditional methods, the number of iterations required is only 12.7%, and it also outperforms other loss functions. This explains why the ReL-GB loss function has a significantly higher accuracy at 100 and 200 iterations, indicating that in this task, ReL-GB can still quickly overcome the impact of data imbalance.

[0122] (3) This experiment demonstrates the performance of the ResNet model on the imbalanced CIFAR-100 image classification task by applying the ReL-GB loss function, CE loss function, ICF-CE loss function, FL function and CBL function respectively.

[0123] As shown in Table 3, regarding the number of iterations required to escape the gradient trap, unlike the first two sets of experiments, the more complex models enabled all loss functions to achieve improved recall of minor class samples with fewer training epochs. ReL-GB and ICF-CE, in particular, began to improve minor class sample recall in the first training epoch in all parallel experiments, still significantly outperforming the control function. In terms of classification accuracy, all loss functions benefited from the ResNet structure, achieving similar accuracy at training completion. However, ReL-GB maintained its leading accuracy across several training stages, indicating its more effective ability to improve model accuracy. Furthermore, ReL-GB boasted the highest F1 score and a recall rate second only to ICF-CE, demonstrating its ability to more effectively extract knowledge of minor class samples from imbalanced datasets.

[0124] Table 3 shows the performance of models with different loss functions on the imbalanced CIFAR-100 classification task.

[0125]

[0126] (4) This experiment demonstrates the performance of the LSTM model on the Yelp dataset text classification task by applying the ReL-GB loss function, CE loss function, ICF-CE loss function, FL function and CBL function respectively.

[0127] Table 4 shows the performance of models using different loss functions on the Yelp dataset classification task.

[0128]

[0129]

[0130] Table 4 shows that, regarding accuracy, the ReL-GB model achieved the best accuracy performance at 100, 200, and 1000 iterations, only briefly being surpassed by the FL model at 500 iterations. This indicates that the ReL-GB loss function remains highly effective for LSTM models and text classification tasks. Similar to other experiments, ICF-CE achieved the highest recall for minor class samples but performed the worst in accuracy and F1 score. This demonstrates that ICF-CE cannot effectively extract information from minor class samples, but merely guesses more samples as minor class samples. Therefore, we believe that ReL-GB, with recall second only to ICF-CE, can more effectively learn minor class sample knowledge in imbalanced data. Furthermore, ReL-GB achieved the best performance compared to the other four loss functions in terms of minor class sample F1 score and training cost for escaping gradient traps.

[0131] In the experiments, various classification models employing the ReL-GB loss function demonstrated excellent performance on binary classification tasks using imbalanced datasets of different data types. Compared to cross-entropy loss, focal loss, and class balance loss weighted by effective sample count, ReL-GB achieved the highest accuracy at both the beginning and end of training, while maintaining the highest F1 score and excellent recall. Furthermore, the ReL-GB model was the fastest to escape gradient traps. This demonstrates its effectiveness in mitigating model training problems caused by imbalanced data. Compared to the widely used cross-entropy function, the ReL-GB model significantly reduces the computational cost required for model convergence. Even compared to the high-performing focal loss and class balance loss weighted by effective sample count, ReL-GB loss also exhibits higher average accuracy and F1 score. The experiments used four different network models for validation, reflecting the universality of the ReL-GB loss function across different models and tasks. Moreover, ReL-GB does not require hyperparameter tuning based on the dataset, which facilitates its widespread application to various training tasks.

[0132] More importantly, ReL-GB's efficient classification of imbalanced datasets improves the performance of the self-growing models discussed later. Our proposed self-growing neural network is trained using a linear perceptron to learn imbalanced binary classification tasks. On the one hand, its training time is significantly affected by the training speed of the imbalanced binary classification sub-tasks; on the other hand, its model performance is also affected by the accuracy of the imbalanced binary classification sub-tasks. In practical applications, the convergence conditions for imbalanced binary classification tasks are extremely stringent. Existing loss functions, such as cross-entropy, struggle to converge within a finite time, while other loss functions cannot efficiently complete training. Therefore, we constructed ReL-GB to ensure the efficient operation of the self-growing network.

[0133] In the early stages of model training, the test accuracy of the model trained by this invention is much higher than that of the other four comparison models, and even higher than the test accuracy of the comparison models after 200 iterations. The model classification accuracy is improved, the gradient update in the early stages of model training is more effective, and the model convergence is accelerated. When training is completed, the model trained by this invention has the highest test accuracy compared to the control model.

[0134] The model trained by this invention is least affected by imbalanced datasets and can get rid of gradient traps and incorrect optimization problems caused by imbalanced data with the fewest number of iterations. The number of iterations consumed from the start of training to the start of learning small class sample information is only 17.3% of that of the traditional cross-entropy loss optimization method.

[0135] Upon completion of training, the model trained by this invention has the highest F1 score for minor class samples compared to the control model, meaning it is able to learn minor class sample knowledge in the dataset most effectively, and the model has higher accuracy and recall for minor class samples.

[0136] Secondly, this invention also proposes a novel classification system for linear rectified gradient balance loss functions, such as... Figure 4 As shown, it includes:

[0137] Function definition module 41: Used to define the linear rectified gradient balancing loss function;

[0138] Model building module 42: Used to build multiple required models and initialize the parameters of the models;

[0139] Dataset Acquisition Module 43: Used to acquire the required dataset;

[0140] Dataset preprocessing module 44: Used to preprocess the acquired dataset;

[0141] Model Iterative Training Module 45: Used to iteratively train the model according to a preset method;

[0142] Comparison Module 46: Used to compare the results of training an LSTM model using cross-entropy loss, weighted cross-entropy, focal loss, and class balance loss with a model trained using the linear rectified gradient balance loss function.

[0143] The following is for reference. Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 The diagram shows the structure of a computer device 600 (a server or terminal device). Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0144] like Figure 5 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 603 or programs loaded from storage section 609 into random access memory (RAM) 606. Various programs and data required for the operation of device 600 are also stored in RAM 604. The CPU 601, GPU 602, ROM 603, and RAM 604 are interconnected via bus 605. Input / output (I / O) interface 606 is also connected to bus 605.

[0145] The following components are connected to I / O interface 606: an input section 607 including a keyboard, mouse, etc.; an output section 608 including an LCD, speakers, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card, such as a LAN card or modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to I / O interface 606 as needed. A removable medium 612, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 611 as needed so that computer programs read from it can be installed into storage section 609 as needed.

[0146] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 610, and / or installed from removable medium 612. When the computer program is executed by central processing unit (CPU) 601 and graphics processing unit (GPU) 602, the functions defined in the methods of the present invention are performed.

[0147] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution apparatus, device, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0148] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0151] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the method described in the first aspect.

[0152] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A linear rectifier gradient balanced loss function classification method, characterized by, The method comprises the following steps: defining a linear rectified gradient balance loss function; constructing a model and initializing parameters of the model; obtaining a data set and preprocessing the data set; preprocessing the data set further comprises: taking user ratings with data labels of 1-2 as negative ratings; taking user ratings with data labels of 3-5 as positive ratings to construct a text binary classification problem; wherein the ratio of the number of large class samples to the number of small class samples is 3.69; iteratively training the model in a predetermined manner, specifically comprising: inputting a batch of samples into the network, calculating the class prediction value output by the forward propagation; for the samples with classification errors, calculating the loss value of the samples using a linear rectified loss function; for the samples with correct classification, calculating the loss value of the samples using a gradient balance loss function; calculating the weighted sum of the gradients of the two loss functions, and the weighting coefficient is a positive constant, wherein the weighting coefficient is set to 1 in the experiment; updating the network parameters using the gradient of the weighted sum; wherein the optimizer of the training network uses the Adam optimizer to perform 1000 rounds of iterative optimization at a learning rate of 0.0001; The function includes a linear rectifier loss function L ic (x), defined as follows: L ic (x) = x · (β(x) - z) wherein x is the classification prediction value output by the sample through the network forward propagation, and z is the sample category label; The function also includes a gradient balance loss function L c (x), when the network employs Sigmoid as the output layer activation function, is defined as follows: wherein σ is the Sigmoid function, w0 and w1 respectively correspond to the sum of the gradients calculated for the samples with labels of 0 and 1, C1 is the logits set of the correctly classified positive class samples in the current training batch, C0 is the logits set of the correctly classified negative class samples in the current training batch, and w0 and w1 do not participate in back propagation.

2. The linear rectifier gradient-balanced loss function classification method of claim 1, wherein, The constructed model is a double-layer LSTM model, wherein each layer of the model contains 64 hidden units.

3. The linear rectifier gradient-balanced loss function classification method of claim 1, wherein, The data set is a Yelp data set, and preprocessing the data set comprises: selecting 150,000 data with text length of 10 to 300 words as the training data set, and 10,000 data with the same length range as the test data set; converting each word into an 8000-dimensional vector through a word2vec algorithm.

4. The linear rectifier gradient-balanced loss function classification method of claim 1, wherein, Further comprising: using the same parameter settings, training the LSTM model using cross-entropy loss, weighted cross-entropy, focal loss and class balance loss respectively under the same data set, and comparing with the model trained by the linear rectified gradient balance loss function.

5. A linear rectifier gradient balanced loss function classification system, characterized by, The system is used to implement the method of any one of claims 1-4, comprising: a function definition module for defining a linear rectified gradient balance loss function; a model construction module for constructing a required model and initializing parameters of the model; a data set acquisition module for acquiring a required data set; a data set preprocessing module for preprocessing the acquired data set; a model iterative training module for iteratively training the model in a predetermined manner; a comparison module for comparing the results of training the LSTM model using cross-entropy loss, weighted cross-entropy, focal loss and class balance loss with the model trained by the linear rectified gradient balance loss function.

6. An electronic device, comprising: one or more processors; storage means for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a method as claimed in any of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, which when executed by a processor, implements a method as claimed in any of claims 1-4.

Images