Reinforcement learning driven unbalanced data classification task optimization method

Abstract

The invention provides an unbalanced data classification task optimization method driven by reinforcement learning, and mainly aims to solve the problems that when minority class samples are few or complex in distribution, synthetic samples may not completely and accurately reflect real distribution and key information is lost. According to the scheme, a confrontation scene is designed and a reinforcement learning model is introduced to assist training, so that the classification precision and the generalization ability of the model can be remarkably improved; the reinforcement learning model is responsible for selecting a proper training sample and adjusting parameters based on a prediction result of the classifier; and meanwhile, the classifier model receives the samples for prediction, and updates the model according to a prediction result. The algorithm provided by the invention has excellent classification precision in the unbalanced cifar-10 data set, is obviously superior to the similar algorithm, and is not limited to any specific model type, so that extremely high flexibility and expandability are provided, and the algorithm has strong application potential.

Description

Technical Field

[0001] The present invention belongs to the field of image classification processing, and mainly relates to a reinforcement learning driven imbalanced data classification task optimization method. Background Art

[0002] Deep learning is an advanced machine learning technique based on a complex multi-layer neural network architecture, which makes it particularly suitable for dealing with complex pattern recognition and data mapping problems. In many fields, especially computer vision and natural language processing, deep learning technology has demonstrated its powerful capabilities, and its performance is even comparable to that of humans in some tasks. With the continuous advancement of technology, applications based on deep learning have developed rapidly and achieved innovation in many fields. However, deep learning is a data-driven algorithm, and its performance depends largely on the quality of the training data set, which limits the performance of deep learning in practical applications with poor training data quality. In real-world machine learning applications, we often encounter the problem of unbalanced data distribution, especially in classification tasks. Data imbalance means that there are far more samples of some categories in the data set than other categories. This unbalanced data distribution will cause the classification model to favor the majority class, thereby ignoring or misclassifying the minority class, affecting the overall classification performance.

[0003] To solve this problem, researchers have developed a variety of methods to optimize the classification task training of imbalanced datasets. One of the current mainstream methods is to balance the category distribution in the dataset by directly modifying the structure of the dataset. The most common strategies include oversampling and undersampling.

[0004] Oversampling techniques such as SMOTE increase the number of minority class samples by synthesizing new samples in their feature space, which helps to enrich feature expression and enhance the generalization ability of the model. However, when the minority class samples themselves are small or the distribution is complex, the synthesized samples may not fully and accurately reflect the true distribution, introduce noise, and affect the training effect. On the other hand, undersampling balances the dataset by randomly reducing the majority class samples, but this method may lead to the loss of key information, especially those samples that are critical to classification decisions, which may weaken the model's ability to capture the key features of the majority class and reduce overall accuracy and reliability. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, when the minority class samples themselves are small or the distribution is complex, the synthesized samples may not fully and accurately reflect the real distribution and key information is lost. The present invention provides a reinforcement learning driven imbalanced data classification task optimization method.

[0006] A reinforcement learning driven imbalanced data classification task optimization method comprises the following steps:

[0007] Step S1: Initialize the data set, build a classifier model, initialize the classifier model parameters, build and initialize the reinforcement learning model;

[0008] Step S2: training the reinforcement learning model;

[0009] Train the reinforcement learning model; calculate the Q value of each possible action based on the state of the reinforcement learning model, and select the action to be executed based on the Q value and the ε-greedy strategy; perform importance sampling according to the priority of the samples in the corresponding data set according to the action, and send the obtained batch data to the classifier model for prediction, and calculate the reward based on the prediction results; store the learned experience in the experience replay pool of the reinforcement learning model, and the reinforcement learning model samples experience from the experience replay pool for learning, and updates the reinforcement learning model parameters;

[0010] Step S3: Update the classifier model;

[0011] The classifier model is updated; the reinforcement learning model takes action according to the state, and performs importance sampling according to the priority parameters of each sample to obtain importance sampling batch data; at the same time, random sampling batch data is obtained by randomly sampling from the training data, and the two are combined into training batch data; the training batch data is input into the classifier model for learning, the parameters of the classifier model are updated, and the priority of the data in the training batch is updated according to the prediction results; after the training is completed, the obtained classifier model is the final result, and the application of this classifier model has the optimal classification accuracy.

[0012] Furthermore, the detailed steps of step S1 are as follows: initializing the data set, building a classifier and initializing the classifier model parameters, and building and initializing the reinforcement learning model:

[0013] Step S1-1: Initialize the data set;

[0014] First, preprocess the dataset;

[0015] The preprocessing method is to adjust the size of all samples in the data set to c×w×h, where c is the number of channels, w is the width, and h is the height to ensure the consistency of the input data;

[0016] Divide the preprocessed data set into training sets {DATA} according to function train , validation set {DATA} validation and the test set {DATA} test ;

[0017] For the training set {DATA} train The samples in are classified according to different labels L = {l1,l2,...,l i ,...,l N} to cluster and obtain N sets {S1,S2,...,S i ,...,S N}, each set S i Contains all labels in the training set l i samples; N is the total number of labels;

[0018] Assign an initial value p0 to the priority parameter p(e) of each sample e in the training set; the initial priority is the basis for the reinforcement learning model to select samples in the subsequent training process. This step ensures that the data can be used correctly and efficiently in the subsequent training;

[0019] Step S1-2: construct a classifier and initialize model parameters;

[0020] The classifier includes an initial convolutional layer, a residual network layer and an output layer;

[0021] The structure of the initial convolution layer is as follows: the number of input channels of the initial convolution layer is 3, the number of output channels of the initial convolution layer is 64, the convolution kernel size is 3x3, the step size is 1, and the padding is 1; the input of the initial convolution layer is the original image data, and preliminary feature extraction is performed on the input sample, and the output is a feature map after preliminary feature extraction; the initial convolution layer is used to extract image features;

[0022] The residual network layer includes a basic unit residual block Res-block; the basic unit residual block Res-block includes a first layer sequence, a second layer sequence, a third layer sequence, and a fourth layer sequence; each layer sequence contains two residual blocks;

[0023] The basic unit residual block Res-block includes Conv1, Bn1, ReLU, Conv2 and Bn2 layers;

[0024] The Conv1 layer and the Conv2 layer have the same structure. The size of the Conv1 layer and the Conv2 layer are both 3x3, the stride is 1, the padding is 1, and the number of channels is 64; the padding of the Bn1 layer and the Bn2 layer is 0, and the number of channels is 64; the number of channels of ReLU is 64;

[0025] The size of the residual block is 1x1, the step size is 1, the padding is zero, and the number of channels is 64;

[0026] The number of channels of the first layer sequence is 64; the number of channels of the second layer sequence is 128; the number of channels of the third layer sequence is 256; the number of channels of the fourth layer sequence is 512;

[0027] The input of the residual network layer is the feature map output by the initial convolutional layer; in this layer group, the feature map is processed by a series of residual blocks, each of which further deepens the feature extraction; the output of the residual network layer is a feature map after complex feature extraction and optimization;

[0028] The input of the output layer is the feature map output by the residual network layer. The output layer sequentially performs average pooling, flattening, and full connection layer mapping operations on the feature map output by the residual network layer. The output of the output layer is a classification prediction result with a dimension of N.

[0029] The input of the classifier is a batch of samples {b1, b2, ..., b i ,…,b k}, b i represents a sample, k represents the number of samples in a batch; the size of each sample is c×w×h;

[0030] Sampling based on the priority parameters of the sample is called importance sampling, and the probability of the sample being sampled is calculated as follows:

[0031]

[0032] Among them, p(e) is the priority corresponding to sample e, and α is the priority weight adjustment factor;

[0033] The output of the classifier is the category to which the predicted sample label belongs The number of categories to which the predicted sample labels belong is the same as the number of classification labels in the training set; this step ensures that the classifier can adapt to different types of data sets and correctly output the category prediction for each instance.

[0034] Furthermore, the dataset adopts CIFAR-10; N=10.

[0035] Furthermore, the steps of training the reinforcement learning model in step S2 are:

[0036] Step 2-1: Initialize the reinforcement learning model state S;

[0037] The state S is a set {S1,S2,...,S i ,...,S N}, the number of times selected; set S i Is a collection for storing classifier data training; set S i For {x1,x2,…,x i ,…,x N} i The number of times selected;

[0038] At the beginning of the round eposide, let S = {0, 0, 0, ..., 0} to represent the initial state of each round, and initialize the number of action steps n = 0;

[0039] Step 2-2: Reinforcement learning model RL_agent action selection;

[0040] The reinforcement learning model receives the state S and calculates the value estimate Q value for each possible action; based on the value estimate Q value, the reinforcement learning model selects action a i to execute;

[0041] The specific steps of action selection of the reinforcement learning model RL_agent are to input the vector representing the state S into the reinforcement learning model and output the value {q1,q2,...q i …,q N}, the reinforcement learning model uses the ε-greedy strategy for action selection, ε is a parameter between 0 and 1, which controls the balance between exploration and utilization, that is, a is selected with a probability of 1-ε i = argmax a q i ; Randomly select actions with probability ε, and finally get the output action a of the reinforcement learning model i ; Each output action a i Corresponding set S i ;

[0042] Step 2-3: Take action a based on the output of the reinforcement learning model i , in the corresponding set S i In the example, importance sampling is performed according to the priority parameter p(e) of each sample to generate batch data {b1,b2,…,b i ,…,b k};

[0043] The probability of each sample being sampled is calculated as follows:

[0044]

[0045] p(e) is the priority corresponding to sample e, α is the priority weight adjustment factor, S i is the corresponding set;

[0046] Step 2-4: The sampled batch samples {b1,b2,…,b i ,…,b k} Input the classifier to make predictions, calculate the reward R based on the prediction results, and update the priority p(e) of the data in the batch;

[0047] The reward R formula is as follows:

[0048]

[0049] β is the scaling factor, k is the total number of samples in the batch, and c is the number of correctly classified samples;

[0050] Update the sample priority according to the prediction results. The calculation formula is as follows:

[0051] p(e)=|δ(e)|+∈

[0052] |δ(e)| is the error between the prediction of sample e and the actual label, ∈ is a positive number less than 0.001;

[0053] Step 2-5: Calculate the next state S' of the reinforcement learning model and determine whether the round eposide has ended. If the round has ended, do = True; if the round has not ended, do = False; set state S and action a i , reward R, next state S′, whether the round is over or not is combined into experience conversion (S, a i ,R,S′,done),(S,a i ,R,S′,done) is stored in the experience replay pool of the reinforcement learning model;

[0054] Step 2-6: The reinforcement learning model samples experience in the experience replay pool to learn and update parameters;

[0055] The reinforcement learning model randomly extracts batches of experience transitions from the experience replay pool {(S i ,a i ,R i ,S′ i ,done i )} for learning and updating, (S i ,a i ,R i ,S′ i ,done i ) represents one of the experience transformations, and the time difference TD algorithm is used to update the reinforcement learning model parameters.

[0056] First, calculate the reinforcement learning model prediction value Q(S i ,a i ; θ) is when the reinforcement learning model parameter is θ, for a given state S i and action a i estimate of the value of;

[0057] Then calculate the TD target r i is the reward value, γ is the discount factor, ranging from 0 to 1, maxQ(S′ i,a;θ) means that when the reinforcement learning model parameter is θ, the next state S′ i Take the maximum value estimate of all possible actions; then calculate the TD error Finally, update the parameter θ of the reinforcement learning model using the following formula:

[0058]

[0059] Where α is the learning rate, In state S i and action a i The value of estimates the gradient with respect to θ;

[0060] Step 2-7: Reinforcement learning model state transition S←S′;

[0061] The current state is S = {x1, x2, ..., x i ,…,x N}, the next state is S′={x1,x2,…,x i +1,…,x N}, transfer the reinforcement learning model from the current state to the next state S = S′;

[0062] If the round is not over, return to step 2-2; if the round is over but the round number eposide has not reached the set maximum round number T, the round number eposide increases by 1 and returns to step 2-1 to start a new round; if the round is over and the round number eposide reaches the set maximum round number T, go to step S3;

[0063] Furthermore, the step S3 of updating the classifier model is as follows:

[0064] Step S3-1: Initialize the environment and state S;

[0065] Step S3-2: The reinforcement learning model takes action a based on the current state S i ;

[0066] Step S3-3: Take action a according to the output of the reinforcement learning model i , in the corresponding set S i In the example, importance sampling is performed according to the priority parameters of each sample to obtain the importance sampling batch data {b1, b2, …, b i ,…,b k};

[0067] Step S3-4: Perform random sampling in the training data to obtain random sampling batch data {c1, c2, ..., c i ,…,c m}, m is the number of random sampling batch data samples, which is combined with the importance sampling batch data {b1,b2,…,b i ,…,b k} combination to obtain the training batch data {c1,c2,…,c i ,…,c m ,b1,b2,…,b i ,…,b k};

[0068] Step 3-5: Input the training batch data into the classifier for learning, update the classifier model parameters, and update the priority of the data in the batch according to the prediction results;

[0069] Input the training batch data into the classifier and get the prediction Using the cross entropy loss function To calculate the prediction The loss with label Y Update the classifier network weights using stochastic gradient descent (SGD) η is the learning rate, which controls the step size of network weight adjustment; is the loss function The gradient of the weight ω; the priority p(e) of updating the data according to the prediction results;

[0070] Step 3-6: Calculate the next state S′ and determine whether the round is over;

[0071] Calculate the next state of the reinforcement learning model S′={x1,x2,…,x i +1,…,x N}, the round action step count increases by 1, and it is determined whether the set total round step count M has been reached. If it has reached M, it means the round is over; if it has not reached M, it means the round is not over;

[0072] Step S3-7: Reinforcement learning model state transition S←S′;

[0073] The current state is S, S = {x1, x2, ..., x i ,…,x N}, the next state is S′, S′={x1,x2,…,x i +1,…,x N}, transfer the reinforcement learning model from the current state to the next state;

[0074] If the round has ended, the classifier model update phase ends. If the round has not ended, return to step 3-2;

[0075] Determine whether the training round epoch has reached the maximum training round K. If it has not reached the maximum training round K, the epoch increases by 1 and returns to the reinforcement learning training stage S2. Otherwise, the training ends.

[0076] The execution of the reinforcement learning model training phase and the classifier model update phase is recorded as one training round epoch. The training ends after K training rounds are executed in a cycle.

[0077] Furthermore, the reinforcement learning model is built based on the DQN algorithm or Q-learning or A2C algorithm.

[0078] Furthermore, the classifier is ResNet-18.

[0079] The present invention proposes a classifier training framework integrating reinforcement learning model. By introducing reinforcement learning agent and designing specific confrontation scenarios, the reinforcement learning agent intelligently adjusts the selection and utilization of samples in the classifier training process, optimizes the training process of the model, and significantly improves the adaptability and processing effect of the classifier to imbalanced data. Compared with the previous imbalanced classification data set processing algorithm, the algorithm proposed in the present invention has better classification accuracy in the imbalanced cifar-10 data set. Compared with SMOTE, the overall classification accuracy is improved by 2.03%, and the classification accuracy of sample scarce classes (class 3 and class 7) is improved by 8.3% and 7.9% respectively; compared with the undersampling method, the overall classification accuracy is improved by 5.56%, and the classification accuracy of sample scarce classes (class 3 and class 7) is improved by 11.3% and 13.6% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the system framework of the present invention;

[0081] Figure 2 It is a distribution diagram of the training data of the present invention in various categories; DETAILED DESCRIPTION

[0082] The reinforcement learning-driven optimization method for imbalanced data classification tasks includes the following steps:

[0083] S1. Initialization phase;

[0084] The initialization phase initializes the data set and reinforcement learning model;

[0085] Step S1-1: Initialize the data set;

[0086] First, preprocess the dataset;

[0087] The preprocessing method is to adjust the size of all samples in the data set to c×w×h, where c is the number of channels, w is the width, and h is the height to ensure the consistency of the input data;

[0088] Divide the preprocessed data set into training sets {DATA} according to function train , validation set {DATA} validation and the test set {DATA} test ;

[0089] For the training set {DATA} train The samples in are classified according to different labels L = {l1,l2,...,l i ,...,l N} to obtain N sets {S1,S2,...,S i ,...,S N}, each set S i Contains all labels in the training set l i samples; N is the total number of labels;

[0090] Assign an initial value p0 to the priority parameter p(e) of each sample e in the training set; the initial priority is the basis for the reinforcement learning model to select samples in the subsequent training process. This step ensures that the data can be used correctly and efficiently in the subsequent training;

[0091] The dataset uses CIFAR-10; N=10;

[0092] The distribution of training data in each category is as follows Figure 2 As shown, the number of samples in category 3 and category 7 is significantly lower than that in other categories;

[0093] Step S1-2: construct a classifier and initialize model parameters;

[0094] The classifier includes an initial convolutional layer, a residual network layer and an output layer;

[0095] The classifier is ResNet-18, and the network parameters are initialized; in the present invention, the output dimension N is set to 10, corresponding to the 10 categories of the CIFAR-10 data set;

[0096] The structure of the initial convolution layer is as follows: the number of input channels of the initial convolution layer is 3, the number of output channels of the initial convolution layer is 64, the convolution kernel size is 3x3, the step size is 1, and the padding is 1; the input of the initial convolution layer is the original image data, and preliminary feature extraction is performed on the input sample, and the output is a feature map after preliminary feature extraction; the initial convolution layer is used to extract image features;

[0097] The residual network layer includes a basic unit residual block Res-block; the basic unit residual block Res-block includes a first layer sequence, a second layer sequence, a third layer sequence, and a fourth layer sequence; each layer sequence contains two residual blocks;

[0098] The basic unit residual block Res-block includes Conv1, Bn1, ReLU, Conv2 and Bn2 layers;

[0099] The Conv1 layer and the Conv2 layer have the same structure. The size of the Conv1 layer and the Conv2 layer are both 3x3, the stride is 1, the padding is 1, and the number of channels is 64; the padding of the Bn1 layer and the Bn2 layer is 0, and the number of channels is 64; the number of channels of ReLU is 64;

[0100] The size of the residual block is 1x1, the step size is 1, the padding is zero, and the number of channels is 64;

[0101] The number of channels of the first layer sequence is 64; the number of channels of the second layer sequence is 128; the number of channels of the third layer sequence is 256; the number of channels of the fourth layer sequence is 512;

[0102] The input of the residual network layer is the feature map output by the initial convolutional layer; in this layer group, the feature map is processed by a series of residual blocks, each of which further deepens the feature extraction; the output of the residual network layer is a feature map after complex feature extraction and optimization;

[0103] The input of the output layer is the feature map output by the residual network layer. The output layer sequentially performs average pooling, flattening, and full connection layer mapping operations on the feature map output by the residual network layer. The output of the output layer is a classification prediction result with a dimension of N.

[0104] The input of the classifier is a batch of samples {b1, b2, ..., b i ,…,b k}, b i represents a sample, k represents the number of samples in a batch; the size of each sample is c×w×h;

[0105] Sampling based on the priority parameters of the sample is called importance sampling, and the probability of the sample being sampled is calculated as follows:

[0106]

[0107] Among them, p(e) is the priority corresponding to sample e, and α is the priority weight adjustment factor;

[0108] The output of the classifier is the category to which the predicted sample label belongs The number of categories to which the predicted sample labels belong is the same as the number of classification labels in the training set; this step ensures that the classifier can adapt to different types of data sets and correctly output the category prediction for each instance;

[0109] The initial convolutional layer structure is shown in the following table:

[0110]

[0111] The structure of the residual block Res-block is as follows:

[0112]

[0113] The link structure between residual blocks is as follows:

[0114]

[0115] The number of channels set in the four layer sequences is different. The residual blocks in the same layer sequence use the same number of channels. The number of channels from the first layer sequence to the fourth layer sequence are as follows:

[0116] Layer1 Layer2 Layer3 Layer4 Number of channels 64 128 256 512

[0117] Step S1-3: Build a reinforcement learning model and initialize parameters;

[0118] Use the DQN algorithm to build a reinforcement learning model;

[0119] The input of the reinforcement learning model is N sets {S1,S2,...,S i ,...,S N The number of times each of them is selected is the set {x1,x2,...,x i …,x N}, x i Represents the set S in the historical training process i The number of times selected;

[0120] The action space of the reinforcement learning model is {a1,a2,…,a i ,…,a N}, action a i Corresponding selection set S i , N = 10;

[0121] The output of the reinforcement learning model is an estimate of the value of each action in the action space {q1,q2,...q i …,q N}, q i Indicates action a i The value estimate of q i The higher the value, the more the reinforcement learning model tends to take action ai , that is, select the corresponding set S i ;

[0122] Initialize the experience replay pool of the reinforcement learning model, store the experience during the reinforcement learning model training process, extract experience learning from it, and update the parameters of the reinforcement learning model;

[0123] Methods for building reinforcement learning models include using Q-learning and A2C algorithms;

[0124] The key to this step is to ensure that the reinforcement learning model can achieve effective data interaction with the classifier so that the reinforcement learning model can adjust its strategy for selecting training samples based on the performance feedback of the classifier; this interaction ensures information synchronization and optimization between the two models, which is a key link in improving training efficiency and model accuracy;

[0125] A reinforcement learning model is built based on the DQN algorithm, and the network parameters in the reinforcement learning model are initialized. In the reinforcement learning model built by the DQN algorithm, the value network Q-Net is used to approximate the Q function, which represents the expected return of selecting an action in a given state.

[0126] The value network Q-Net includes three first fully connected layers, a second fully connected layer, and a third fully connected layer; the input dimension of the first fully connected layer is 10, the output dimension is 128, and the activation function is ReLU; the input dimension of the second fully connected layer is 128, the output dimension is 128, and the activation function is ReLU; the input dimension of the third fully connected layer is 128, and the output dimension is 10;

[0127] Use three fully connected layers to build the value network Q-Net. The value network is used to approximate the value function in reinforcement learning. The structure is as follows:

[0128] Layer Number Input Dimensions Output Dimensions Activation Function 1 10 128 ReLU 2 128 128 ReLU 3 128 10 -

[0129] In the present invention, in order to express this information, we use a one-dimensional vector containing 10 elements, each element records the number of times the corresponding set is selected, so the input state size is 10; the output of the reinforcement learning model is the value estimate of each possible action, that is, the Q value; in the invention, the data set contains 10 categories, and accordingly, the neural network will output a vector with a dimension of 10, in which each element represents the action value estimate of the corresponding category;

[0130] S2. Reinforcement learning model training stage;

[0131] In this stage, the classifier parameters are frozen and only predictions are performed. The reinforcement learning model is trained to guide it to find the weaknesses of the classifier.

[0132] Step 2-1: Initialize the reinforcement learning model state S;

[0133] The state S is a set {S1,S2,...,S i ,...,S N}, the number of times selected; set S i Is a collection for storing classifier data training; set S i For {x1,x2,…,x i ,…,x N} i The number of times selected;

[0134] At the beginning of the round eposide, let S = {0, 0, 0, ..., 0} to represent the initial state of each round, and initialize the number of action steps n = 0;

[0135] Step 2-2: Reinforcement learning model RL_agent action selection;

[0136] The reinforcement learning model receives the state S and calculates the value estimate Q value for each possible action; based on the value estimate Q value, the reinforcement learning model selects action a i to execute;

[0137] The specific steps of action selection of the reinforcement learning model RL_agent are to input the vector representing the state S into the reinforcement learning model and output the value {q1,q2,...q i ...,q N}, the reinforcement learning model uses the ε-greedy strategy for action selection, ε is a parameter between 0 and 1, which controls the balance between exploration and utilization, that is, a is selected with a probability of 1-ε i = argmax a q i ; Randomly select actions with probability ε, and finally get the output action a of the reinforcement learning model i ; Each output action a i Corresponding set S i ;

[0138] Step 2-3: Take action a based on the output of the reinforcement learning model i , in the corresponding set S i In the example, importance sampling is performed according to the priority parameter p(e) of each sample to generate batch data {b1,b2,…,b i ,…,b k};

[0139] The probability of each sample being sampled is calculated as follows:

[0140]

[0141] p(e) is the priority corresponding to sample e, α is the priority weight adjustment factor, S i is the corresponding set;

[0142] Step 2-4: The sampled batch samples {b1,b2,…,b i ,…,b k} Input the classifier to make predictions, calculate the reward R based on the prediction results, and update the priority p(e) of the data in the batch;

[0143] The reward R formula is as follows:

[0144]

[0145] β is the scaling factor, k is the total number of samples in the batch, and c is the number of correctly classified samples;

[0146] Update the sample priority according to the prediction results. The calculation formula is as follows:

[0147] p(e)=|δ(e)|+∈

[0148] |δ(e)| is the error between the prediction of sample e and the actual label, ∈ is a positive number less than 0.001;

[0149] Step 2-5: Calculate the next state S' of the reinforcement learning model and determine whether the round eposide has ended. If the round has ended, do = True; if the round has not ended, do = False; set state S and action a i , reward R, next state S′, whether the round is over or not is combined into experience conversion (S, a i ,R,S′,done),(S,a i ,R,S′,done) is stored in the experience replay pool of the reinforcement learning model;

[0150] Step 2-6: The reinforcement learning model samples experience in the experience replay pool to learn and update parameters;

[0151] The reinforcement learning model randomly extracts batches of experience transitions from the experience replay pool {(S i ,a i ,R i ,S′ i ,done i )} for learning and updating, (S i ,a i ,R i ,S′ i ,done i ) represents one of the experience transformations, and the time difference TD algorithm is used to update the reinforcement learning model parameters.

[0152] First, calculate the reinforcement learning model prediction value Q(S i ,a i ; θ) is when the reinforcement learning model parameter is θ, for a given state S i and action a i estimate of the value of;

[0153] Then calculate the TD target r i is the reward value, γ is the discount factor, ranging from 0 to 1, maxQ(S′ i ,a;θ) means that when the reinforcement learning model parameter is θ, the next state S′ i Take the maximum value estimate of all possible actions; then calculate the TD error Finally, update the parameter θ of the reinforcement learning model using the following formula:

[0154]

[0155] Where α is the learning rate, In state S i and action a i The value of estimates the gradient with respect to θ;

[0156] Step 2-7: Reinforcement learning model state transition S←S′;

[0157] The current state is S = {x1, x2, ..., x i ,…,x N}, the next state is S′={x1,x2,…,x i +1,…,x N}, transfer the reinforcement learning model from the current state to the next state S = S′;

[0158] If the round is not over, return to step 2-2; if the round is over but the round number eposide has not reached the set maximum round number T, the round number eposide increases by 1 and returns to step 2-1 to start a new round; if the round is over and the round number eposide reaches the set maximum round number T, go to step S3;

[0159] S3. Classifier model update stage: In this stage, the parameters of the reinforcement learning model are frozen and only sample sampling is performed. The classifier model is mainly trained based on the data sampled by the reinforcement learning model to improve its classification performance.

[0160] Step S3-1: Initialize the environment and state S;

[0161] Step S3-2: The reinforcement learning model takes action a based on the current state Si ;

[0162] Step S3-3: Take action a according to the output of the reinforcement learning model i , in the corresponding set S i In the example, importance sampling is performed according to the priority parameters of each sample to obtain the importance sampling batch data {b1, b2, …, b i ,…,b k};

[0163] Step S3-4: Perform random sampling in the training data to obtain random sampling batch data {c1, c2, ..., c i ,…,c m}, m is the number of random sampling batch data samples, which is combined with the importance sampling batch data {b1,b2,…,b i ,…,b k} combination to obtain the training batch data {c1,c2,…,c i ,…,c m ,b1,b2,…,b i ,…,b k};

[0164] Step 3-5: Input the training batch data into the classifier for learning, update the classifier model parameters, and update the priority of the data in the batch according to the prediction results;

[0165] Input the training batch data into the classifier and get the prediction Using the cross entropy loss function To calculate the prediction The loss with label Y Update the classifier network weights using stochastic gradient descent (SGD) η is the learning rate, which controls the step size of network weight adjustment; is the loss function The gradient of the weight ω; the priority p(e) of updating the data according to the prediction results;

[0166] Step 3-6: Calculate the next state S′ and determine whether the round is over;

[0167] Calculate the next state of the reinforcement learning model S′={x1,x2,…,x i +1,…,x N}, the round action step count increases by 1, and it is determined whether the set total round step count M has been reached. If it has reached M, it means the round is over; if it has not reached M, it means the round is not over;

[0168] Step S3-7: Reinforcement learning model state transition S←S′;

[0169] The current state is S, S = {x1, x2, ..., x i ,…,x N}, the next state is S′, S′={x1,x2,…,x i +1,…,x N}, transfer the reinforcement learning model from the current state to the next state;

[0170] If the round has ended, the classifier model update phase ends. If the round has not ended, return to step 3-2;

[0171] Determine whether the training round epoch has reached the maximum training round K. If it has not reached the maximum training round K, the epoch increases by 1 and returns to the reinforcement learning training stage S2. Otherwise, the training ends.

[0172] The execution of the reinforcement learning model training phase and the classifier model update phase is recorded as one training round epoch. The training ends after K training rounds are executed in a cycle.

[0173] The present invention will be further described below in conjunction with embodiments and drawings.

[0174] The design ideas of the present invention are as follows:

[0175] The purpose of this invention is to propose a classifier training framework that integrates a reinforcement learning model. By introducing a reinforcement learning agent and designing a specific adversarial scenario, it aims to intelligently adjust the selection and utilization of samples during the classifier training process through the reinforcement learning agent, optimize the model training process, and effectively improve the model's ability to process and learn imbalanced data sets.

[0176] (1) Adversarial scenario design;

[0177] Scenario modeling: The reinforcement learning agent is only responsible for data sampling and passing it to the classifier for prediction and training. By modeling the data sampling-assisted training process as a Markov decision process (MDP), the system's ability to control the training dynamics is increased. In this process, each decision depends not only on the current state, but also on the subsequent state generated from it, forming a decision chain. In this scenario, the reinforcement learning agent and the classifier are in an adversarial relationship. The agent will always choose data that is difficult for the classifier to correctly classify in order to obtain rewards. This mechanism forces the classifier to continuously adjust and optimize its parameters to adapt to difficult-to-predict samples, thereby improving the overall training efficiency and the robustness of the model.

[0178] Reward setting: This design encourages the reinforcement learning model to select data samples that may cause the classifier to make wrong predictions through a carefully designed reward mechanism. For example, if a sample is misclassified, the reinforcement learning model will receive a positive reward, otherwise it will be punished or have a lower reward. According to the greedy rule, the reinforcement learning agent will be more inclined to find samples that can make the classifier wrong in order to maximize the reward.

[0179] (2) Intelligent sample selection mechanism;

[0180] Importance sampling and priority adjustment: This framework introduces an intelligent sample selection mechanism based on importance sampling. A dynamic priority is set for each training sample, which is adjusted according to the prediction results of the sample during the classification process. Specifically, the priority of samples with incorrect predictions will be increased, so that these samples will be selected more frequently in subsequent training; conversely, the priority of samples with correct predictions will be reduced. This mechanism ensures that the focus is on difficult samples during training, especially in the case of data imbalance, and can effectively improve the recognition ability of minority samples.

[0181] Dynamically adjust strategies: The reinforcement learning model dynamically adjusts the selection and use strategies of training samples according to the performance of the classifier during training and the overall learning progress. This not only optimizes training efficiency, but also helps improve the adaptability and accuracy of the model in practical applications.

[0182] The method of the present invention is described by taking cifar-10 as the data set, resnet-18 as the classifier model, and DQN algorithm as the reinforcement learning algorithm as an example.

[0183] The method of the present invention is described by taking cifar-10 as the data set, resnet-18 as the classifier model, and DQN algorithm as the reinforcement learning algorithm as an example. Figure 1 As shown;

[0184] 1. Initialization phase

[0185] Step 1-1: Divide cifar-10 into training sets {Img} by category train , validation set, test set. Divide them according to the label category and give each piece of data an initial priority. The distribution of training data in each category is as follows Figure 2 As shown, the number of samples in category 3 and category 7 is significantly lower than that in other categories.

[0186] Step 1-2: Build a classifier based on ResNet-18 and initialize the network parameters.

[0187] This classifier consists of three main parts, the details of which are as follows:

[0188] (1) Initial convolutional layer: used to extract image features. The structure is as follows:

[0189]

[0190] (2) Residual block and layer combination: The core of this part is composed of the basic unit residual block (Res-block), which is further organized into different layer sequences (layers). This part consists of four layer sequences, each layer contains two residual blocks. Taking the first layer as an example, the structure of a single residual block is as follows:

[0191]

[0192] Its residual link (short) structure is as follows:

[0193]

[0194] Different layers have different channel numbers. Different residual blocks of the same layer use the same number of channels. The channel numbers from the first layer to the fourth layer are:

[0195] Layer1 Layer2 Layer3 Layer4 Number of channels 64 128 256 512

[0196] (3) Output layer: The output layer is the last part of the model, responsible for converting the features learned from the previous layers into the final classification results. This part mainly consists of an average pooling layer and a fully connected layer. The average pooling layer uses a 4x4 window to pool the output of the last convolutional layer to reduce the spatial dimension of the feature map; then the pooled multi-dimensional data is converted into a one-dimensional array through a flattening operation so that it can be input into the fully connected layer; in the specific parameters of the fully connected layer, the input dimension depends on the output of the previous layer, which is usually the flattened feature vector; the output dimension is 10, corresponding to the 10 categories of the CIFAR-10 dataset.

[0197] Step 1-3: Build a reinforcement learning agent based on the DQN algorithm and initialize the agent network parameters. Use three fully connected layers to build a value network (Q-Net) for value function approximation in reinforcement learning. Its structure is as follows:

[0198] Layer Number Input Dimensions Output Dimensions Activation Function 1 10 128 ReLU 2 128 128 ReLU 3 128 10 -

[0199] The input of the reinforcement learning model neural network is the number of times each category of the training data is selected. In this experiment, in order to express this information, we use a one-dimensional vector containing 10 elements, each element records the number of times the corresponding category is selected, so the input state size is 10. The output of the network is an estimate of the value of each possible action, that is, the Q value. In this experiment, the data set contains 10 categories, and accordingly, the neural network will output a vector of dimension 10, where each element represents the action value estimate of the corresponding category.

[0200] 2. Reinforcement learning model training phase

[0201] Step 2-1: Initialize the environment, the number of action steps n and the agent state S. The agent state is represented by {x0, x1, x2, …, x9}, where x i Indicates the number of times each category is selected. At the beginning of the round (eposide), let S = {0, 0, 0, ..., 0} to represent the initial state of each round, and let n = 0.

[0202] Step 2-2: Input the vector representing the state S into the neural network and output the value Q(S,a) corresponding to each action. i ), the agent uses the ε-greedy strategy to select actions, where ε is a parameter between 0 and 1 that controls the balance between exploration and exploitation. That is, it selects a with a probability of 1-ε. * = argmax a Q(s,a); select any action a with probability ε. Get the action A selected by the agent.

[0203] Step 2-3: Perform importance sampling according to the category corresponding to action A. The probability of each sample being sampled is calculated as follows:

[0204]

[0205] p(e) is the priority corresponding to sample e, and α is the priority weight adjustment factor.

[0206] Step 2-4: Input the sampled batch data into the classifier for prediction, and calculate the reward based on the prediction results. The formula is as follows:

[0207]

[0208] β is the scaling factor, n is the total number of samples in the batch, and c is the number of correctly classified samples.

[0209] Update the sample priority according to the result. The calculation formula is as follows:

[0210] p(e)=|δ(e)|+∈

[0211] |δ(e)| is the error between the prediction of sample e and the actual label, ∈ is a very small positive number.

[0212] Step 2-5: Calculate the next state S′={x0,x1,…,x A +1,…,x9},n=n+1, and determine whether the set total number of round steps N has been reached. Store (S,A,R,S′,done) in the agent’s experience replay pool.

[0213] Step 2-6: Randomly extract batches of experience from the experience replay pool {(S i ,A i ,R i ,S′ i ,done i )}Use the TD algorithm to update the agent parameters. First calculate the DQN prediction value Then calculate the TD target γ·maxQ(s t+1 ,a;θ), and then calculate the TD error Finally, update the DQN parameters using the following formula:

[0214]

[0215] Step 2-7: Agent state transition S←S′.

[0216] If the round has not ended, return to step 2-2. If the round has ended but the maximum number of rounds has not been reached, return to step 2-1. If the round has ended and the maximum number of rounds has been reached, this stage ends.

[0217] 3. Classifier model update phase

[0218] Step 3-1: Initialize the environment and agent state S.

[0219] Step 3-2: The reinforcement learning agent (RL_agent) takes action A based on the current state S.

[0220] Step 3-3: Perform importance sampling according to the category corresponding to action A.

[0221] Step 3-4: Randomly sample the training data and combine it with the data obtained from the previous importance sampling to generate batch data {Img} train .

[0222] Step 3-5: Input batch data into the classifier and get predictions The cross entropy loss function is used to calculate the loss between the prediction and the label Y: Update the classifier network weights using stochastic gradient descent (SGD)

[0223] Step 3-6: Calculate the next state S′={x0,x1,…,x A +1,…,x9},n=n+1, and judge whether the number of steps n has reached the set total number of steps N of the round.

[0224] Step 3-7: Agent state transition S←S′.

[0225] If the total number of steps N is not reached, return to step 3-2, otherwise this stage ends.

[0226] Determine whether the maximum epoch has been reached. If not, epoch = epoch + 1 and return to the reinforcement learning training stage. Otherwise, the training ends.

[0227] The present invention proposes a classifier training framework integrating reinforcement learning model. By introducing reinforcement learning agent and designing specific confrontation scenarios, the reinforcement learning agent intelligently adjusts the selection and utilization of samples in the classifier training process, optimizes the training process of the model, and significantly improves the adaptability and processing effect of the classifier to imbalanced data. Compared with the previous imbalanced classification data set processing algorithm, the algorithm proposed in the present invention has better classification accuracy in the imbalanced cifar-10 data set. Compared with SMOTE, the overall classification accuracy is improved by 2.03%, and the classification accuracy of sample scarce classes (class 3 and class 7) is improved by 8.3% and 7.9% respectively; compared with the undersampling method, the overall classification accuracy is improved by 5.56%, and the classification accuracy of sample scarce classes (class 3 and class 7) is improved by 11.3% and 13.6% respectively.

Claims

1. A reinforcement learning driven imbalanced data classification task optimization method, characterized in that: The steps include: Step S1: Initialize the data set, build a classifier model, initialize the classifier model parameters, build and initialize the reinforcement learning model; Step S2: training the reinforcement learning model; Train the reinforcement learning model; calculate the Q value of each possible action based on the state of the reinforcement learning model, and select the action to be executed based on the Q value and the ε-greedy strategy; perform importance sampling according to the priority of the samples in the corresponding data set according to the action, and send the obtained batch data to the classifier model for prediction, and calculate the reward based on the prediction results; The learned experience is stored in the experience replay pool of the reinforcement learning model. The reinforcement learning model samples experience from the experience replay pool for learning and updates the reinforcement learning model parameters. Step S3: Update the classifier model; Update the classifier model; The reinforcement learning model takes actions according to the state and performs importance sampling according to the priority parameters of each sample to obtain importance sampling batch data. At the same time, random sampling is performed from the training data to obtain random sampling batch data, and the two are combined into training batch data. Input the training batch data into the classifier model for learning, update the parameters of the classifier model, and update the priority of the data in the training batch according to the prediction results; After the training is completed, the obtained classifier model is the final result, and the application of this classifier model has the best classification accuracy.

2. The method for optimizing imbalanced data classification tasks driven by reinforcement learning according to claim 1, characterized in that: The detailed steps of step S1 are as follows: initializing the data set, building a classifier model and initializing the classifier model parameters, and building and initializing the reinforcement learning model: Step S1-1: Initialize the data set; First, preprocess the dataset; The preprocessing method is to adjust the size of all samples in the data set to c×w×h, where c is the number of channels, W is the width, and h is the height to ensure the consistency of the input data; Divide the preprocessed data set into training sets {DATA} according to function train , validation set {DATA} validation and the test set {DATA} test ; For the training set {DATA} train The samples in are classified according to different labels L = {l1, l2, ..., l i , ..., l N } to obtain N sets {S1, S2, ..., S i , ..., S N }, each set S i Contains all labels in the training set l i samples; N is the total number of labels; Assign an initial value p0 to the priority parameter p(e) of each sample e in the training set; Step S1-2: construct a classifier model and initialize model parameters; The classifier model includes an initial convolutional layer, a residual network layer and an output layer; The structure of the initial convolution layer is as follows: the number of input channels of the initial convolution layer is 3, the number of output channels of the initial convolution layer is 64, the convolution kernel size is 3x3, the step size is 1, and the padding is 1; the input of the initial convolution layer is the original image data, and preliminary feature extraction is performed on the input sample, and the output is a feature map after preliminary feature extraction; the initial convolution layer is used to extract image features; The residual network layer includes a basic unit residual block Res-block; the basic unit residual block Res-block includes a first layer sequence, a second layer sequence, a third layer sequence, and a fourth layer sequence; each layer sequence contains two residual blocks; The basic unit residual block Res-block includes Conv1, Bn1, ReLU, Conv2 and Bn2 layers; the Conv1 layer and the Conv2 layer have the same structure, the size of the Conv1 layer and the Conv2 layer are both 3x3, the step size is 1, the padding is 1, and the number of channels is 64; the padding of the Bn1 layer and the Bn2 layer is 0, the number of channels is 64; the number of channels of the ReLU is 64; The size of the residual block is 1x1, the step size is 1, the padding is zero, and the number of channels is 64; The number of channels of the first layer sequence is 64; the number of channels of the second layer sequence is 128; the number of channels of the third layer sequence is 256; the number of channels of the fourth layer sequence is 512; The input of the residual network layer is the feature map after the initial feature extraction output by the initial convolution layer; the output of the residual network layer is the feature map after complex feature extraction and optimization; The input of the output layer is the feature map output by the residual network layer. The output layer sequentially performs average pooling, flattening, and full connection layer mapping operations on the feature map output by the residual network layer. The output of the output layer is a classification prediction result with a dimension of N. The input of the classifier model is a batch of samples {b1, b2, ..., b i ,…,b k }, b i represents a sample, k represents the number of samples in a batch; the size of each sample is c×w×h; Sampling based on the priority parameters of the sample is called importance sampling, and the probability of the sample being sampled is calculated as follows: Among them, p(e) is the priority corresponding to sample e, and α is the priority weight adjustment factor; The output of the classifier model is the category to which the predicted sample label belongs The number of categories to which the predicted sample labels belong is the same as the number of classification labels in the training set.

3. The method for optimizing imbalanced data classification tasks driven by reinforcement learning according to claim 1, characterized in that: The steps of step S2 for training the reinforcement learning model are: Step 2-1: Initialize the reinforcement learning model state S; The state S is a set {S1, S2, ..., S i , ..., S N }The number of times it is selected; Set S i For {x1, x2, ..., x i , ..., x N } i The number of times selected; At the beginning of the round eposide, let S = {0, 0, 0, ..., 0} to represent the initial state of each round, and initialize the number of action steps n = 0; Step 2-2: Reinforcement learning model RL_agent action selection; The reinforcement learning model receives the state S and calculates the value estimate Q value of each possible action; based on the value estimate Q value, the reinforcement learning model selects action a i to execute; The specific steps of action selection of the reinforcement learning model RL_agent are to input the vector representing the state S into the reinforcement learning model and output the value {q1, q2, ...q i ..., q N }, the reinforcement learning model uses the ε-greedy strategy for action selection, ε is a parameter between 0 and 1, that is, a is selected with a probability of 1-ε i = argmax a q i ; Randomly select actions with probability ε, and finally get the output action a of the reinforcement learning model i ; Each output action a i Corresponding set S i ; Step 2-3: Take action a based on the output of the reinforcement learning model i , in the corresponding set S i In the example, importance sampling is performed according to the priority parameter p(e) of each sample to generate batch data {b1, b2, ..., b i , ..., b k }; The probability of each sample being sampled is calculated as follows: p(e) is the priority corresponding to sample e, α is the priority weight adjustment factor, S i is the corresponding set; Step 2-4: The sampled batch samples {b1, b2, ..., b i , ..., b k } Input the classifier model to make predictions, calculate the reward R based on the prediction results, and update the priority p(e) of the data in the batch; The reward R formula is as follows: β is the scaling factor, k is the total number of samples in the batch, and c is the number of correctly classified samples; Update the sample priority according to the prediction results. The calculation formula is as follows: p(e)=|δ(e)|+∈ |δ(e)| is the error between the prediction of sample e and the actual label, ∈ is a positive number less than 0.001; Step 2-5: Calculate the next state S' of the reinforcement learning model and determine whether the round eposide has ended. If the round has ended, set done = True; if the round has not ended, set done = False; set state S and action a i , reward R, next state S′, whether the round is over, done, and the combination is converted into experience (S, a i ,R,S′,done), will (S,a i , R, S′, done) are stored in the experience replay pool of the reinforcement learning model; Step 2-6: The reinforcement learning model samples experience in the experience replay pool to learn and update parameters; The reinforcement learning model randomly extracts batches of experience transitions from the experience replay pool {(S i , a i , R i , S′ i , done i )} for learning and updating, (S i , a i , R i , S′ i , done i ) represents one of the experience transformations, and the time difference TD algorithm is used to update the reinforcement learning model parameters; First, calculate the reinforcement learning model prediction value Q(S i , a i ; θ) is when the reinforcement learning model parameter is θ, for a given state S i and action a i Estimated value of; Then calculate the TD target r i is the reward value, γ is the discount factor, ranging from 0 to 1, maxQ(S′ i , a; θ) means that when the reinforcement learning model parameter is θ, the next state S′ i Take the maximum value estimate of all possible actions; then calculate the TD error Finally, update the parameter θ of the reinforcement learning model using the following formula: Where α is the learning rate, In state S i and action a i The value of estimates the gradient with respect to θ; Step 2-7: Reinforcement learning model state transition S←S′; The current state is S = {x1, x2, ..., x i , ..., x N }, the next state is S′={x1,x2,...,x i +1, ..., x N }, transfer the reinforcement learning model from the current state to the next state S = S′; If the round has not ended, return to step 2-2; if the round has ended but the number of rounds eposide has not reached the set maximum number of rounds T, the number of rounds eposide increases by 1 and returns to step 2-1 to start a new round; if the round has ended and the number of rounds eposide has reached the set maximum number of rounds T, enter step S3.

4. The method for optimizing imbalanced data classification tasks driven by reinforcement learning according to claim 1, characterized in that: The steps of updating the classifier model in step S3 are: Step S3-1: Initialize the environment and state S; Step S3-2: The reinforcement learning model takes action a based on the current state S i ; Step S3-3: Take action a according to the output of the reinforcement learning model i , in the corresponding set S i In the example, importance sampling is performed according to the priority parameters of each sample to obtain the importance sampling batch data {b1, b2, ..., b i , ..., b k }; Step S3-4: Perform random sampling in the training data to obtain random sampling batch data {c1, c2, ..., c i , ..., c m }, m is the number of random sampling batch data samples, which is combined with the importance sampling batch data {b1, b2, ..., b i , ..., b k } combination to obtain the training batch data {c1, c2, ..., c i , ..., c m , b1, b2, ..., b i , ..., b k }; Step 3-5: Input the training batch data into the classifier model for learning, update the classifier model parameters, and update the priority of the data in the batch according to the prediction results; Input the training batch data into the classifier model to get the prediction Using the cross entropy loss function To calculate the prediction The loss with label Y Update the classifier model network weights using stochastic gradient descent (SGD) ηη is the learning rate, which controls the step size of network weight adjustment; is the loss function The gradient of the weight ω; the priority p(e) of updating the data according to the prediction results; Step 3-6: Calculate the next state S′ and determine whether the round is over; Calculate the next state of the reinforcement learning model S′={x1,x2,...,x i +1, ..., x N }, the round action step count increases by 1, and it is determined whether the set round total step count M has been reached. If it has reached M, it means the round is over; if it has not reached M, it means the round is not over; Step S3-7: Reinforcement learning model state transition S ← S′; The current state is S, S = {x1, x2, ..., x i , ..., x N }, the next state is S′, S′={x1,x2,...,x i +1, ..., x N }, transfer the reinforcement learning model from the current state to the next state; If the round has ended, the classifier model update phase ends. If the round has not ended, return to step 3-2; Determine whether the training round epoch has reached the maximum training round K. If it has not reached the maximum training round K, the epoch increases by 1 and returns to the reinforcement learning training stage S2. Otherwise, the training ends. The execution of the reinforcement learning model training phase and the classifier model update phase is recorded as one training round epoch. The training ends after K training rounds are executed in a cycle.

5. The method for optimizing imbalanced data classification tasks driven by reinforcement learning according to claim 1, characterized in that: The data set uses CIFAR-10; N=10.

6. The method for optimizing imbalanced data classification tasks driven by reinforcement learning according to claim 1, characterized in that: The classifier model is ResNet-18.

7. The method for optimizing imbalanced data classification tasks driven by reinforcement learning according to claim 1, characterized in that: The reinforcement learning model is constructed based on the DQN algorithm or Q-learning or A2C algorithm.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, a reinforcement learning-driven imbalanced data classification task optimization method according to any one of claims 1 to 7 is implemented.

Images