Data set classification method and system based on unbalanced incremental learning

Abstract

The invention discloses a data set classification method and system based on unbalanced incremental learning, and the method comprises the steps: obtaining an original data set, and carrying out the preprocessing of the original data set, and obtaining a training set and a test set; determining an unbalanced incremental learning model, and performing model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model; and obtaining a current to-be-processed data set, inputting the current to-be-processed data set into the data set classification model, and outputting a data set classification result. According to the method, the non-equilibrium incremental learning model is trained and optimized to obtain the data set classification model, and the classification efficiency and accuracy of the data set can be effectively improved through the data set classification model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data set classification method, system, terminal and computer-readable storage medium based on unbalanced incremental learning. Background Art

[0002] With the development of the times, dynamic data environments are widely present in various application scenarios. Data not only flows in continuously, but is also often accompanied by the problem of category imbalance.

[0003] Traditional data processing models are unable to effectively deal with the problem of dynamic changes in categories in data sets, especially in unbalanced data scenarios, where model performance often degrades and the ability to recognize minority data is weak, resulting in low classification efficiency of data sets and inaccurate classification results.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a data set classification method, system, terminal and computer-readable storage medium based on unbalanced incremental learning, aiming to solve the problem that traditional data processing models in the prior art are difficult to effectively cope with the dynamic changes of categories in the data set, resulting in low classification efficiency of the data set and inaccurate classification results.

[0006] To achieve the above object, the present invention provides a data set classification method based on unbalanced incremental learning, and the data set classification method based on unbalanced incremental learning comprises the following steps:

[0007] Obtaining an original data set, and preprocessing the original data set to obtain a training set and a test set;

[0008] Determine an unbalanced incremental learning model, and perform model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model;

[0009] The current data set to be processed is obtained, and the current data set to be processed is input into the data set classification model, and the data set classification result is output.

[0010] Optionally, the data set classification method based on unbalanced incremental learning, wherein the obtaining of the original data set and preprocessing of the original data set to obtain a training set and a test set, specifically includes:

[0011] Acquire an original data set, and perform data cleaning on the original data set to obtain a first data set;

[0012] Determine whether the first data set is a classification feature, and if so, use one-hot encoding to perform numerical conversion processing on the first data set to obtain a second data set;

[0013] The second data set is divided into a training set and a test set.

[0014] Optionally, in the data set classification method based on unbalanced incremental learning, the step of determining whether the first data set is a classification feature further includes:

[0015] If the first data set is not a categorical feature, determining whether the first data set is a numerical feature;

[0016] If so, normalizing the first data set using a standardization tool to obtain a third data set;

[0017] The third data set is divided into a training set and a test set.

[0018] Optionally, the data set classification method based on unbalanced incremental learning, wherein the unbalanced incremental learning model includes a variational autoencoder and an incremental adaptive adjustment classifier;

[0019] The determining of the unbalanced incremental learning model, and performing model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model specifically includes:

[0020] Inputting the training set into the variational autoencoder in the unbalanced incremental learning model, performing feature extraction processing on the training set through the variational autoencoder to obtain the mean and variance of the latent variables;

[0021] Inputting the mean and variance of the latent variable into the incremental adaptive adjustment classifier, and performing classification training processing on the mean and variance of the latent variable by the incremental adaptive adjustment classifier to obtain an initial data set classification model;

[0022] Performing a model performance evaluation process on the initial data set classification model through the test set to obtain a model performance evaluation result, wherein the model performance evaluation process includes classification performance evaluation, residual analysis, robustness test, and feature importance evaluation;

[0023] The initial data set classification model is optimized according to the model performance evaluation result to obtain a data set classification model.

[0024] Optionally, in the data set classification method based on unbalanced incremental learning, the variational autoencoder includes an encoder and a decoder;

[0025] The step of inputting the training set into the variational autoencoder in the unbalanced incremental learning model, performing feature extraction processing on the training set by the variational autoencoder to obtain the mean and variance of the latent variables specifically includes:

[0026] Inputting the training set into the encoder in the variational autoencoder, and performing data mapping processing and gradient descent optimization processing on the training set by the encoder to obtain latent variables;

[0027] Inputting the latent variables into the decoder, and performing remapping processing on the latent variables through the decoder to obtain reconstructed data;

[0028] Calculating the reconstruction error and relative entropy between the training set and the reconstructed data, and weighting the reconstruction error and the relative entropy to obtain an overall loss function;

[0029] The variational autoencoder is optimized according to the overall loss function to obtain an optimized variational autoencoder, and the mean and variance of the latent variables are output.

[0030] Optionally, the data set classification method based on non-balanced incremental learning, wherein the mean and variance of the latent variable are input into the incremental adaptive adjustment classifier, and the mean and variance of the latent variable are subjected to classification training processing by the incremental adaptive adjustment classifier to obtain an initial data set classification model, specifically includes:

[0031] Performing sample weight initialization processing on the mean and variance of the latent variable to obtain an initialized sample weight;

[0032] Determine a plurality of preset first classifiers, and train the plurality of preset first classifiers according to the initialization sample weights to obtain a plurality of second classifiers;

[0033] Calculating a weighted error rate of each of the second classifiers on the training set, and calculating a classifier weight of each of the second classifiers according to the weighted error rate;

[0034] Acquire classification results of a plurality of the second classifiers, and update the classifier weights according to the classification results and the weighted error rate to obtain an optimized incremental adaptive adjustment classifier;

[0035] The initial data set classification model is obtained through the optimized variational autoencoder and the optimized incremental adaptive adjustment classifier.

[0036] Optionally, the data set classification method based on unbalanced incremental learning, wherein the step of obtaining the current data set to be processed, inputting the current data set to be processed into the data set classification model, and outputting the data set classification result, specifically includes:

[0037] Acquire a current data set to be processed, and input the current data set to be processed into the data set classification model;

[0038] The data set classification model is used to perform academic warning analysis on the current data set to be processed to obtain a data set classification result.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also provides a data set classification system based on unbalanced incremental learning, wherein the data set classification system based on unbalanced incremental learning includes:

[0040] A preprocessing module is used to obtain an original data set and preprocess the original data set to obtain a training set and a test set;

[0041] A model training module, used to determine an unbalanced incremental learning model, and perform model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model;

[0042] The model output module is used to obtain the current data set to be processed, input the current data set to be processed into the data set classification model, and output the data set classification result.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a data set classification program based on unbalanced incremental learning stored in the memory and run on the processor, and when the data set classification program based on unbalanced incremental learning is executed by the processor, the steps of the data set classification method based on unbalanced incremental learning as described above are implemented.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data set classification program based on unbalanced incremental learning, and when the data set classification program based on unbalanced incremental learning is executed by a processor, the steps of the data set classification method based on unbalanced incremental learning as described above are implemented.

[0045] In the present invention, an original data set is obtained, and the original data set is preprocessed to obtain a training set and a test set; an unbalanced incremental learning model is determined, and model training and model optimization are performed on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model; a current data set to be processed is obtained, and the current data set to be processed is input into the data set classification model, and a data set classification result is output. The present invention obtains a training set and a test set by preprocessing the original data set, and trains and optimizes the unbalanced incremental learning model by the training set and the test set to obtain a data set classification model. The data set classification model can improve the recognition ability of a small number of samples, and at the same time, it can also quickly and accurately analyze the data set, effectively improving the classification efficiency and accuracy of the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of a preferred embodiment of the data set classification method based on unbalanced incremental learning of the present invention;

[0047] Figure 2 It is a schematic diagram of the overall structure of a preferred embodiment of the data set classification method based on non-balanced incremental learning of the present invention;

[0048] Figure 3 It is a schematic diagram of the overall implementation process of a preferred embodiment of the data set classification method based on non-balanced incremental learning of the present invention;

[0049] Figure 4 It is a schematic diagram of analyzing the student academic warning situation of a preferred embodiment of the data set classification method based on non-balanced incremental learning of the present invention;

[0050] Figure 5 It is a structural diagram of a preferred embodiment of a data set classification system based on unbalanced incremental learning of the present invention;

[0051] Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] With the development of the times, dynamic data environments are widely present in various application scenarios. Whether in medical, educational or industrial process control environments, data not only flows in continuously, but is also often accompanied by the problem of category imbalance.

[0054] Traditional data processing is difficult to effectively deal with the problem of dynamic changes in categories in data streams, especially in unbalanced data scenarios, where model performance often degrades or the ability to identify minority classes is weak. At the same time, information loss is also a common challenge in dynamic data processing, especially in a constantly changing data stream. How to keep the memory of old data and efficiently learn new data has become a difficult problem that needs to be solved urgently.

[0055] In view of the above problems, the present invention proposes a data set classification method based on unbalanced incremental learning, which performs efficient data extraction through a variational autoencoder to ensure that information is not lost during the conversion process. At the same time, the dynamic adaptability and adaptive weight adjustment mechanism of the incremental Adaboost classifier are combined to ensure that the recognition ability of minority class data is improved when processing unbalanced data sets. The present invention is a method based on the combination of a variational autoencoder and an incremental Adaboost classifier, which can effectively improve the processing effect of unbalanced data sets.

[0056] The data set classification method based on unbalanced incremental learning described in the preferred embodiment of the present invention is as follows: Figure 1 He Ru Figure 2 As shown, the data set classification method based on unbalanced incremental learning includes the following steps:

[0057] Step S10: Obtain an original data set, and preprocess the original data set to obtain a training set and a test set.

[0058] After obtaining the original data set, the present invention will perform data cleaning and preprocessing on the original data set, wherein it is necessary to perform feature discrimination on the original data set to determine whether it is a classification feature, a numerical feature, or other features, and then divide it into a training data set and a test set of the model.

[0059] Specifically, an original data set is obtained, and data cleaning is performed on the original data set to obtain a first data set; it is determined whether the first data set is a classification feature, and if so, the first data set is numerically converted using one-hot encoding to obtain a second data set; the second data set is partitioned to obtain a training set and a test set.

[0060] Furthermore, if the first data set is not a categorical feature, it is determined whether the first data set is a numerical feature; if so, a standardization tool is used to normalize the first data set to obtain a third data set; and the third data set is divided into a training set and a test set.

[0061] like Figure 3As shown, after obtaining the original data set, the original data set needs to be cleaned and preprocessed first. The preprocessing process is as follows: For the disordered categorical features in the data (such as gender, name, etc., which are disordered features), one-hot encoding is used for numerical conversion; for the ordered categorical features in the data, the features are normalized using standardization tools to ensure that each feature participates in training at the same scale. Afterwards, the data set is divided into a training set and a test set (i.e., the training set and the test set in the present invention) in a ratio of 8:2.

[0062] Step S20, determine an unbalanced incremental learning model, and perform model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model, wherein the unbalanced incremental learning model includes a variational autoencoder and an incremental adaptive adjustment classifier.

[0063] In a dynamic data environment, the problem of information retention and category balance is a key bottleneck in existing data processing methods. Due to the continuous addition of new data, traditional methods often face the problem of information loss when performing feature extraction and conversion. The information loss problem not only affects the overall performance of the model, but also leads to a weakening of the memory of the old data categories, and it is impossible to balance the impact of new and old data. In addition, unbalanced data sets are particularly evident in real-time data streams. Minority class samples are easily ignored due to their small number, causing the model to be biased towards the majority class. Therefore, the present invention, by combining a variational autoencoder and an incremental Adaboost classifier, proposes a solution that can dynamically adjust the learning strategy and weight mechanism, which can ensure that information is maximized during feature generation and conversion, and improve sensitivity to minority class samples through real-time weight adjustment.

[0064] It is understandable that category imbalance and information loss are the most critical issues in a dynamic data environment, and it is more difficult to improve model performance than in a static data environment. When faced with the evolution of categories and dynamic changes in feature distribution in data streams, the prior art suffers from problems such as low efficiency, insufficient adaptability, and lack of ability to identify minority classes. Based on this, the present invention provides a data processing method for the problems of category imbalance and information loss in a dynamic data environment. By dynamically adjusting the learning strategy, it can effectively improve the adaptability and stability of the model in the data stream, while taking into account the effective identification of minority classes.

[0065] like Figure 2 As shown, the present invention aims to solve the problems of category imbalance and information loss in dynamic data environments by combining a variational autoencoder (VAE) with an incremental Adaboost classifier (i.e., the incremental adaptive adjustment classifier in the present invention), thereby achieving efficient adaptation to the dynamic evolution of categories in data streams.

[0066] The present invention is divided into the following three parts:

[0067] 1. Feature extraction stage: Use variational autoencoders to extract and optimize data features, and use the latent space of the variational autoencoder to reduce the dimension and compress information on the data to ensure that the key information in the original data is retained to the maximum extent during the feature generation process. At the same time, the self-attention mechanism is used to enhance the global correlation representation ability between features.

[0068] 2. Dynamic classification stage: Introduce the incremental Adaboost (adaptive boosting algorithm) classifier to classify the extracted features, and use the dynamic sample weighting mechanism to gradually update the model parameters according to the category changes in the data stream to enhance the model's adaptability to new categories. At the same time, by calculating the sample weights, balance the distribution of samples of different categories in the data set, and optimize the classification performance of the model on unbalanced data.

[0069] 3. Real-time adjustment stage: During the classification process, by dynamically analyzing the changes in the category distribution in the current data stream and combining the adaptive weight adjustment mechanism, the weight ratio of different categories in the loss function is adjusted to avoid the model being overly biased towards the majority class. On this basis, the robustness test of the model using perturbation analysis can effectively verify the stability of the model when facing input data fluctuations or noise.

[0070] In addition, the present invention further combines residual analysis, learning curve drawing and feature importance evaluation to comprehensively analyze and optimize the performance of the model. Through the combination of dynamic incremental learning strategy and non-balanced optimization mechanism, the present invention significantly improves the adaptability and generalization ability of the model in a dynamic data flow environment, while improving the recognition effect of minority classes.

[0071] Specifically, the training set is input into the encoder in the variational autoencoder, and the encoder performs data mapping processing and gradient descent optimization processing on the training set to obtain latent variables; the latent variables are input into the decoder, and the decoder performs remapping processing on the latent variables to obtain reconstructed data; the reconstruction error and relative entropy between the training set and the reconstructed data are calculated, and the reconstruction error and the relative entropy are weighted to obtain an overall loss function; the variational autoencoder is optimized according to the overall loss function to obtain an optimized variational autoencoder, and the mean and variance of the latent variables are output.

[0072] Feature extraction principle: In a dynamic data environment, feature distribution often changes over time, and traditional methods are difficult to complete feature dimensionality reduction and optimization while maintaining information integrity. The present invention uses a variational autoencoder to perform preliminary feature extraction on the input data, uses the encoder to generate the mean and variance of the data, and maps the data to the latent space (LatentSpace). The features in the latent space have compactness and information integrity, which helps to reduce the redundant information of the input data. In addition, the self-attention mechanism is embedded in the encoding process, so that the model can capture the global dependencies between input features, further optimizing the feature expression.

[0073] like Figure 2 He Ru Figure 3 As shown in the figure, in the feature extraction stage, the present invention uses a variational autoencoder (VAE) to reduce the dimensionality of the input data. The encoder module of the VAE generates a latent space representation of the data to ensure efficient compression and retention of feature information. At the same time, the self-attention mechanism is embedded to model the global dependency between features, further optimizing the feature representation.

[0074] In the specific implementation process, the input data is first received. Here, the input data is preferably a data set (of course, it can also be other data sets, selected according to actual needs), and then the data set is sent to the encoder of the variational autoencoder. The encoder is a neural network that can map the input data to the parameters of the latent space. Assuming that the input data is a vector x, the task of the encoder is to learn the distribution of a latent space. It is usually assumed that this latent space obeys a normal distribution. The output of the encoder is the mean μ and variance σ of the latent space. These parameters describe the probability distribution of the latent variables, and the output of the encoder is: μ = f μ (x) and σ=f σ (x).

[0075] In order to effectively perform gradient descent optimization, VAE uses a reparameterization technique to sample from a random variable that follows a latent distribution, without having to sample directly from a complex distribution, so that the entire process can be back-propagated and gradients can be calculated. Assume that the latent variable z follows a normal distribution N = (μ, σ 2), the reparameterization technique expresses the latent variable z as follows: Z = μ + σ·∈ where ∈~N = (0, I) is a standard normal distribution of random noise, which means sampling a random variable from a standard normal distribution with a mean of 0 and a variance of 1. Where ∈ is a random variable, usually used as noise, introduced in the reparameterization technique to sample from a standard normal distribution to help calculate the gradient, ~ means "obey", that is, the value of ∈ is sampled from a specified probability distribution, N (0, I) is a multivariate normal distribution, 0: represents a vector with a mean of zero (the mean of each dimension is 0), I is a unit covariance matrix, the different dimensions in the distribution are independent of each other, and the variance of each dimension is 1. In this way, z becomes differentiable and can be optimized by gradient descent.

[0076] Further, the latent variable z is passed to the decoder, which is also a neural network that maps the variable z in the latent space back to the data space to generate new data with a similar structure to the original input data. Specifically, the decoder generates reconstructed data through the following process Here, g(z) is the neural network function of the decoder, which maps the latent variable z to a space of the same dimension as the input data.

[0077] The training goal of VAE is to minimize the difference between the input data and the reconstructed data, while ensuring that the distribution of the latent space is close to a standard normal distribution. The training process of VAE consists of two main parts: reconstruction error and KL divergence. The reconstruction error is used to measure the difference between the input data and the reconstructed data, usually measured by mean square error (MSE) or cross entropy. The goal of the reconstruction error is to make the generated samples as close to the original input samples as possible. KL divergence is used to measure the difference between the latent variable distribution q(z|x) and the prior distribution p(z). KL divergence is usually used to limit the distribution of the latent space to be close to the standard normal distribution N(0,I), thereby avoiding the latent space distribution being too complex. The role of this term is to enable the model to generate samples with better generalization ability. The expression of KL divergence is: L KL =D KL (q(z|x)||p(z)), where L KL Represents the value of KL divergence, used as part of the loss function, D KL is the symbol of the divergence, which is used to measure the difference between the two distributions. The overall loss function of VAE is the weighted sum of the reconstruction error and the KL divergence: L VAE =L reconstruction +L KL , L VAE is the overall loss function of VAE, L reconstruction is the reconstruction error, L KLis the KL divergence. During training, VAE can update the parameters of the model by optimizing this overall loss function.

[0078] After training is completed, VAE can be used to generate new data. For example, in the image generation task, VAE generates new images by sampling the standard normal distribution z~N(0,I) of the latent space and then inputting the sample z into the decoder. The generated new images are generated based on the latent variables sampled from the latent space, which is similar to the distribution of the training data.

[0079] Embedding the self-attention mechanism in VAE is usually done by adding a self-attention layer to the encoder or decoder. After the input features pass through several convolutional layers, a self-attention module is added, which is used to calculate the relationship between each feature and perform a weighted combination of the input features based on these relationships to generate an enhanced feature representation.

[0080] The process of self-attention in the VAE encoder is as follows: 1. The input data passes through the convolutional layer in the encoder. 2. Through the self-attention layer, the global dependency of the input features is calculated to obtain the weighted feature representation. 3. Based on the weighted feature representation, the mean μ and variance σ of the latent variable are generated. 4. The latent variable z is sampled from the latent space using the reparameterization technique.

[0081] The process of self-attention in the VAE decoder is as follows: 1. The decoder starts with the latent variable z and recovers the data through a series of deconvolutions. 2. A self-attention layer is added to model the global dependencies between features during the recovery process. 3. By optimizing the reconstruction error and KL divergence, the reconstructed data is generated as similar as possible to the input data.

[0082] like Figure 2 and Figure 3 As shown, after the feature extraction stage is completed, the dynamic classification stage is entered: the present invention introduces an incremental Adaboost (adaptive boosting algorithm) classifier to classify the extracted features, and adopts a dynamic sample weighting mechanism to gradually update the model (i.e., incremental Adaboost classifier model) parameters according to the category changes in the data stream, thereby enhancing the model's adaptability to new category data. At the same time, by calculating sample weights, the distribution of samples of different categories in the data set is balanced, and the classification performance of the model on unbalanced data is optimized.

[0083] After the feature extraction phase is completed, the real-time adjustment phase begins: During the classification process, the weight ratio of different categories in the loss function is adjusted by dynamically analyzing the changes in the category distribution in the current data stream and combining the adaptive weight adjustment mechanism to avoid the model being overly biased towards the majority class. On this basis, the model is tested for robustness using perturbation analysis to verify the stability of the model in the face of input data fluctuations or noise.

[0084] Specifically, the mean and variance of the latent variable are initialized with sample weights to obtain initialized sample weights; multiple preset first classifiers are determined, and the multiple preset first classifiers are trained according to the initialized sample weights to obtain multiple second classifiers; the weighted error rate of each second classifier on the training set is calculated, and the classifier weight of each second classifier is calculated according to the weighted error rate; the classification results of the multiple second classifiers are obtained, and the classifier weights are updated according to the classification results and the weighted error rate to obtain an optimized incremental adaptive adjustment classifier; the initial data set classification model is obtained by the optimized variational autoencoder and the optimized incremental adaptive adjustment classifier;

[0085] The dynamic classification principle set in the present invention is as follows: the incremental Adaboost classifier is used in the present invention to process the features extracted by VAE. Adaboost constructs a series of weak classifiers. Here, the present invention selects linear regression model, decision tree model, SVM model, XGBoost model and LightGBM model as weak classifiers (that is, multiple preset first classifiers in the present invention) to make full use of the advantages of each model. The comparison and analysis of the advantages and disadvantages of the above models are shown in Table 1.

[0086] Table 1: Comparison and analysis of the advantages and disadvantages of each model

[0087]

[0088] Furthermore, the present invention dynamically adjusts the weight of each classifier according to the classification error rate to form a strong classifier (linear regression model, decision tree model, SVM model, XGBoost model and LightGBM model are selected as weak classifiers, and the best strong classifier is constructed by adjusting the different parameters of each model, that is, the optimized incremental adaptive adjustment classifier in the present invention). In an environment where the data stream changes dynamically, the present invention combines the incremental learning method to gradually update the classifier model to adapt to the newly added data categories, while retaining the memory of the old data categories. This dynamic adjustment mechanism ensures that both the majority and minority classes in the data can be effectively identified through sample weighting (Sample Weight).

[0089] It can be understood that in the classification stage, the incremental Adaboost classifier is used to classify the potential features extracted by VAE. The classifier combines the dynamic sample weighting mechanism to deal with the class imbalance problem. By calculating the sample weights, the contribution of minority and majority class samples in the model training process is adaptively adjusted. In addition, the classifier model is gradually updated through the incremental learning method to adapt to the new category distribution in the dynamic data flow environment.

[0090] The specific implementation process of classification processing is as follows: 1. Initialize sample weights: At the beginning of each round of Adaboost iteration, the weights of all training samples are initialized to equal values. The expression for initializing the training sample weights is: w i Initialize the weights for the training samples, for all samples i, where N is the total number of training samples to ensure that each sample contributes equally to the training of the model in the initial stage.

[0091] The specific process of training weak classifiers is as follows: 1. Select weak classifiers: In each round, select a weak classifier (linear regression model, decision tree, SVM, XGBoost or LightGBM) and train the classifier using the current sample weights. Each weak classifier is a simple model that usually performs poorly, but by integrating multiple weak classifiers, a powerful classifier can be obtained. 2. Training process: Each weak classifier will be trained based on the current sample weights and trained using a weighted data set. The model will pay more attention to samples that are misclassified during training. 3. Calculate the weighted error rate of the weak classifier: After training the weak classifier, calculate its weighted error rate on the current sample set. The weighted error rate measures the performance of the classifier under given sample weights. The specific calculation method is as follows: Among them, ∈ t is the weighted error rate of the weak classifier in the tth round, h(x i ) is the weak classifier of the tth round for sample x i The prediction of i is the sample x i The true label, w i is the sample x i 4. Calculate the weight of the weak classifier: The weight of each weak classifier α t According to its weighted error rate ε t The weight of the weak classifier is adjusted according to its performance in the current iteration. The lower the error rate, the greater the weight of the classifier. The calculation formula for the weight of the weak classifier is: According to this formula, if the classifier's error rate is less than 0.5, α t If the error rate is greater than or equal to 0.5, it means that the classifier has a greater contribution in the weighted combination. If the error rate is greater than or equal to 0.5, it means that the classifier performs poorly and the weight will be negative. At this time, further training or adjustment can be used to optimize its effect. 5. Update sample weights: Update the weight of each training sample based on the classification results and error rate of the current weak classifier. The weight of the sample will increase when it is misclassified and decrease when it is correctly classified. The specific update formula is as follows: i (t+1) =w i (t)·exp(1-α t ·y i ·h(x i )), where w i (t) is the weight of sample i in round t, α t is the weight of the weak classifier in the tth round, y i is the true label of sample i, h(x i ) is the prediction of the weak classifier for sample i in the tth round, and the updated sample weight w i (t+1) It indicates the importance of the sample in the next round of training. If a sample is misclassified, its weight will increase, which means that the sample will be given a higher priority the next time the classifier is trained. Conversely, the weight of the correctly classified sample will decrease. 6. The steps of incremental learning are as follows: 1. Dynamically update sample weights: Each time data flows in, the sample weights are updated according to the misclassification of the new data; 2. Update weak classifiers: Incremental Adaboost will train new weak classifiers based on new data and updated sample weights. Gradually adjust the weights of weak classifiers so that they can adapt to the new data distribution. 3. Keep the memory of old data: Even if new data arrives, incremental Adaboost still keeps the memory of old data, and ensures the balance of new and old data through the sample weighting mechanism. 7. Weighted combination of weak classifiers: After each round of iteration, Adaboost will perform a weighted combination of multiple weak classifiers to form the final strong classifier. The final classification result is determined by the weighted prediction results of all weak classifiers. The final classification result formula is as follows: Among them, H(x) is the final classification result, T is the total number of weak classifiers (number of iterations), and h t (x) is the prediction of the weak classifier for sample x in the tth round, α t is the weight of the weak classifier in the tth round. The final classification decision is determined by the weighted results of all weak classifiers. The integrated model makes decisions based on the prediction results of multiple weak classifiers, thereby improving the classification performance.

[0092] In summary, the specific classification process of the features by the incremental Adaboost classifier can be summarized into the following steps: 1. Initialize sample weights: The weights of all samples are initialized to be equal. 2. Train weak classifiers: In each round of iteration, select a weak classifier (linear regression, decision tree, SVM, XGBoost or LightGBM) and train it. 3. Calculate error rate and weight: Calculate the weight of the classifier based on its error rate. 4. Update sample weights: Update the weight of the sample according to the performance of the classifier, and increase the weight of the misclassified sample. 5. Incremental learning: When new data arrives, update the classifier based on the existing model instead of retraining the model. 6. Weighted combination of weak classifiers: The final strong classifier is formed by weighted combination of the outputs of all weak classifiers.

[0093] The initial data set classification model is subjected to model performance evaluation processing through the test set to obtain a model performance evaluation result, wherein the model performance evaluation processing includes classification performance evaluation, residual analysis, robustness test and feature importance evaluation; the initial data set classification model is subjected to model optimization processing according to the model performance evaluation result to obtain a data set classification model.

[0094] like Figure 3 As shown in the figure, after the model training is completed, the performance of the model needs to be evaluated (model performance is evaluated on the test set), mainly including the following aspects: 1. Classification performance: Evaluate the classification effect of the model on the test set through indicators such as accuracy, precision, recall, and F1-score. 1. Residual analysis: Visualize the deviation between the predicted result and the true value to evaluate the prediction accuracy of the model. 3. Robustness test: Add disturbances of different intensities to the test set to verify the model's adaptability to data fluctuations and noise. 4. Feature importance evaluation: Through the feature importance analysis provided by Adaboost, explore which features contribute the most to the classification task.

[0095] The present invention comprehensively analyzes and optimizes the performance of the model by combining residual analysis, learning curve drawing and feature importance evaluation. By combining a dynamic incremental learning strategy with an unbalanced optimization mechanism, the adaptability and generalization ability of the model are significantly improved in a dynamic data flow environment, while improving the recognition effect on minority classes.

[0096] Step S30: Obtain the current data set to be processed, input the current data set to be processed into the data set classification model, and output the data set classification result.

[0097] Specifically, a current data set to be processed is obtained, and the current data set to be processed is input into the data set classification model; the current data set to be processed is subjected to academic warning analysis and processing by the data set classification model to obtain a data set classification result.

[0098] For example, in the early warning analysis process of student academic data, the early warning results include pass or fail, but the student academic data is fluid (continuously updated) in the overall data set, and the academic data with a warning result of fail belongs to a small number of samples. Traditional data processing models are difficult to effectively deal with the problem of dynamic changes in categories in student academic data, especially in unbalanced data scenarios, and often experience a decline in model performance and weak recognition capabilities for minority data, resulting in low efficiency in early warning analysis of student academic data and inaccurate analysis results.

[0099] like Figure 4 As shown in the figure, after the data set classification model is constructed, the current data set to be processed that needs to be analyzed for early warning analysis (here, student academic data is preferably used in the present invention) can be input into the constructed model to analyze the student academic early warning situation (the early warning results are divided into pass and fail, where the student academic data corresponding to the fail result is a minority class sample, therefore, the data set classification result constructed by the present invention can accurately and quickly complete this warning analysis), and the obtained data is as follows: Among them, category 0 has 104 samples (i.e., majority class samples), category 1 has only 20 samples (i.e., minority class samples), the overall accuracy is 0.95, the macro average F1 value is 0.91, and the weighted average F1 value is 0.95, indicating that the overall performance of the model is good, and the recognition ability of the minority class can be further optimized by data enhancement or adjusting the classification threshold.

[0100] This paper proposes a dataset classification method based on unbalanced incremental learning. This method can dynamically adapt to datasets in different fields (without limiting the scope of the dataset and having universal applicability), solve the problem of unbalanced category distribution and severe forgetting in the incremental learning process, and is divided into three parts: data processing, model optimization, and evaluation, and each part supports user-defined extensions. After verification by experimental results, this method performs well on multiple datasets, proving its versatility and scalability.

[0101] The present invention uses a variational autoencoder to perform preliminary feature extraction on the data to ensure maximum information retention during feature generation and representation conversion. Secondly, an incremental Adaboost classifier is used to classify the extracted features, while dynamically adapting to newly added data categories and gradually updating the model to cope with the changing feature distribution and data flow. Finally, in combination with an adaptive weight adjustment mechanism, the weights of different categories in the loss function are dynamically adjusted by real-time analysis of category changes in the current data flow, which can ensure that the model will not be overly biased towards minority classes when processing unbalanced data sets and retain good recognition capabilities for minority classes.

[0102] In summary, in response to the problems of category imbalance and information loss faced by existing data processing methods in dynamic data environments, the present invention proposes a new data processing method based on unbalanced incremental learning. This method can effectively cope with the category evolution in the data stream by dynamically adjusting the learning strategy, and solves the difficulty of low efficiency of traditional data processing methods when facing constantly changing data streams. This method combines the variational autoencoder and the incremental Adaboost classifier, which not only improves the adaptability of the model to new data, but also optimizes the memory retention of old data categories.

[0103] In general, the present invention combines a variety of advanced machine learning techniques to not only improve the efficiency of processing unbalanced data sets, but also enhance the robustness of the model in the face of dynamic changes in data streams. It has efficient learning capabilities and excellent category balance processing effects.

[0104] The present invention has a wide range of market applications, especially in real-time data analysis scenarios. In addition to student academic warning scenarios, it can also be applied to real-time medical monitoring and abnormal behavior monitoring. Its significance lies in that it solves the problems of low efficiency and information loss of traditional methods in dynamic data environments, and provides a more efficient and reliable solution for data processing and decision-making in real-time systems. By enhancing the recognition ability of minority classes, the present invention has far-reaching market value and application prospects in improving the accuracy and stability of the model.

[0105] Furthermore, if Figure 5 As shown, based on the above-mentioned data set classification method based on unbalanced incremental learning, the present invention also provides a data set classification system based on unbalanced incremental learning, wherein the data set classification system based on unbalanced incremental learning includes:

[0106] A preprocessing module 51 is used to obtain an original data set and preprocess the original data set to obtain a training set and a test set;

[0107] A model training module 52 is used to determine an unbalanced incremental learning model, and perform model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model;

[0108] The model output module 53 is used to obtain the current data set to be processed, input the current data set to be processed into the data set classification model, and output the data set classification result.

[0109] Furthermore, if Figure 6 As shown, based on the above-mentioned data set classification method and system based on unbalanced incremental learning, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0110] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a data set classification program 40 based on unbalanced incremental learning is stored on the memory 20, and the data set classification program 40 based on unbalanced incremental learning can be executed by the processor 10, thereby realizing the data set classification method based on unbalanced incremental learning in the present application.

[0111] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the data set classification method based on unbalanced incremental learning.

[0112] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0113] In one embodiment, when the processor 10 executes the data set classification program 40 based on unbalanced incremental learning in the memory 20, the steps of the data set classification method based on unbalanced incremental learning as described above are implemented.

[0114] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data set classification program based on unbalanced incremental learning, and when the data set classification program based on unbalanced incremental learning is executed by a processor, the steps of the data set classification method based on unbalanced incremental learning as described above are implemented.

[0115] In summary, the present invention provides a data set classification method, system, terminal and computer-readable storage medium based on unbalanced incremental learning, the method comprising: obtaining an original data set, and preprocessing the original data set to obtain a training set and a test set; determining an unbalanced incremental learning model, and performing model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model; obtaining a current data set to be processed, and inputting the current data set to be processed into the data set classification model, and outputting a data set classification result. The present invention obtains a training set and a test set by preprocessing the original data set, and trains and optimizes the unbalanced incremental learning model by the training set and the test set to obtain a data set classification model. The data set classification model can improve the recognition ability of a small number of samples, and at the same time, can also quickly and accurately analyze the data set, effectively improving the classification efficiency and accuracy of the data set.

[0116] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0117] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0118] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A data set classification method based on non-balanced incremental learning, characterized in that: The data set classification method based on unbalanced incremental learning includes: Obtaining an original data set, and preprocessing the original data set to obtain a training set and a test set; Determine an unbalanced incremental learning model, and perform model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model; The current data set to be processed is obtained, and the current data set to be processed is input into the data set classification model, and the data set classification result is output.

2. The data set classification method based on non-balanced incremental learning according to claim 1 is characterized in that: The obtaining of the original data set and preprocessing the original data set to obtain a training set and a test set specifically includes: Acquire an original data set, and perform data cleaning on the original data set to obtain a first data set; Determine whether the first data set is a classification feature, and if so, use one-hot encoding to perform numerical conversion processing on the first data set to obtain a second data set; The second data set is divided into a training set and a test set.

3. The data set classification method based on non-balanced incremental learning according to claim 2 is characterized in that: The step of determining whether the first data set is a classification feature further includes: If the first data set is not a categorical feature, determining whether the first data set is a numerical feature; If so, normalizing the first data set using a standardization tool to obtain a third data set; The third data set is divided into a training set and a test set.

4. The data set classification method based on non-balanced incremental learning according to claim 1 is characterized in that: The unbalanced incremental learning model includes a variational autoencoder and an incremental adaptive adjustment classifier; The determining of the unbalanced incremental learning model, and performing model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model specifically includes: Inputting the training set into the variational autoencoder in the unbalanced incremental learning model, performing feature extraction processing on the training set through the variational autoencoder to obtain the mean and variance of the latent variables; Inputting the mean and variance of the latent variable into the incremental adaptive adjustment classifier, and performing classification training processing on the mean and variance of the latent variable by the incremental adaptive adjustment classifier to obtain an initial data set classification model; Performing a model performance evaluation process on the initial data set classification model through the test set to obtain a model performance evaluation result, wherein the model performance evaluation process includes classification performance evaluation, residual analysis, robustness test, and feature importance evaluation; The initial data set classification model is optimized according to the model performance evaluation result to obtain a data set classification model.

5. The data set classification method based on non-balanced incremental learning according to claim 4 is characterized in that: The variational autoencoder includes an encoder and a decoder; The step of inputting the training set into the variational autoencoder in the unbalanced incremental learning model, performing feature extraction processing on the training set by the variational autoencoder to obtain the mean and variance of the latent variables specifically includes: Inputting the training set into the encoder in the variational autoencoder, and performing data mapping processing and gradient descent optimization processing on the training set by the encoder to obtain latent variables; Inputting the latent variables into the decoder, and performing remapping processing on the latent variables through the decoder to obtain reconstructed data; Calculating the reconstruction error and relative entropy between the training set and the reconstructed data, and weighting the reconstruction error and the relative entropy to obtain an overall loss function; The variational autoencoder is optimized according to the overall loss function to obtain an optimized variational autoencoder, and the mean and variance of the latent variables are output.

6. The data set classification method based on non-balanced incremental learning according to claim 5 is characterized in that: The step of inputting the mean and variance of the latent variable into the incremental adaptive adjustment classifier, and performing classification training processing on the mean and variance of the latent variable by the incremental adaptive adjustment classifier to obtain an initial data set classification model specifically includes: Performing sample weight initialization processing on the mean and variance of the latent variable to obtain an initialized sample weight; Determine a plurality of preset first classifiers, and train the plurality of preset first classifiers according to the initialization sample weights to obtain a plurality of second classifiers; Calculating a weighted error rate of each of the second classifiers on the training set, and calculating a classifier weight of each of the second classifiers according to the weighted error rate; Acquire classification results of a plurality of the second classifiers, and update the classifier weights according to the classification results and the weighted error rate to obtain an optimized incremental adaptive adjustment classifier; The initial data set classification model is obtained through the optimized variational autoencoder and the optimized incremental adaptive adjustment classifier.

7. The data set classification method based on non-balanced incremental learning according to claim 1 is characterized in that: The step of obtaining a current data set to be processed, inputting the current data set to be processed into the data set classification model, and outputting a data set classification result specifically includes: Acquire a current data set to be processed, and input the current data set to be processed into the data set classification model; The data set classification model is used to perform academic warning analysis on the current data set to be processed to obtain a data set classification result.

8. A data set classification system based on non-balanced incremental learning, characterized in that: The data set classification system based on unbalanced incremental learning includes: A preprocessing module is used to obtain an original data set and preprocess the original data set to obtain a training set and a test set; A model training module, used to determine an unbalanced incremental learning model, and perform model training and model optimization on the unbalanced incremental learning model according to the training set and the test set to obtain a data set classification model; The model output module is used to obtain the current data set to be processed, input the current data set to be processed into the data set classification model, and output the data set classification result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a data set classification program based on unbalanced incremental learning stored in the memory and executable on the processor. When the data set classification program based on unbalanced incremental learning is executed by the processor, the steps of the data set classification method based on unbalanced incremental learning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data set classification program based on unbalanced incremental learning. When the data set classification program based on unbalanced incremental learning is executed by a processor, the steps of the data set classification method based on unbalanced incremental learning as described in any one of claims 1 to 7 are implemented.

Images