Multi-agent reinforcement learning classification method for unbalanced data

Abstract

The invention relates to the technical field of machine learning and data mining, in particular to an unbalanced data-oriented multi-agent reinforcement learning classification method, which comprises the following steps of: firstly, introducing a Markov decision process to construct a classification model based on multi-agent reinforcement learning, the environment provides feedback to the intelligent agent according to the classification result, so that the intelligent agent can learn the unbalanced sample; secondly, different rewards are set for the majority class samples and the minority class samples respectively, a dual deep neural network is used for calculating accumulated rewards, the training process is played back and optimized through priority experience, and learning of the minority class is improved; and finally, intelligent agent centralized training and decentralized execution are combined with distributed calculation, and intelligent agent interaction and learning efficiency is optimized. Compared with an existing unbalanced data classification method, the method has the advantages that the classification accuracy of minority classes can be effectively improved, and the multi-agent cooperative classification time is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning and data mining, and particularly relates to a multi-agent reinforcement learning classification method for imbalanced data. Background Art

[0002] In the fields of machine learning and data mining, the problem of imbalanced data sets refers to the situation where the number of data samples in some classes is far less than that in other classes. This uneven data distribution may cause the model to be biased towards predicting the classes with a larger number of samples during the training process, thus affecting the performance and generalization ability of the model. Especially in practical applications such as fraud detection, disease prediction, and fault diagnosis, the accurate identification of minority class samples is often more important.

[0003] For imbalanced data classification, existing methods can be divided into two categories. Data-level methods include undersampling and oversampling, which balance the number of samples in different classes by adjusting the sample distribution of the data set. Oversampling achieves balance by increasing the number of minority class samples, while undersampling does so by reducing the number of majority class samples. However, data-level methods may lead to overfitting. Especially in the case of oversampling, the repeated generation of minority class samples may make the model overly sensitive to these samples. Undersampling may lose a large amount of useful majority class information, affecting the overall performance of the model. Algorithm-level methods modify the learning algorithm itself, such as using a weighted loss function to give higher weights to minority class samples, or adopting an ensemble method to improve the recognition ability of minority classes. However, algorithm-level methods may increase the complexity of the model, resulting in longer training time or requiring more computing resources. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-agent reinforcement learning classification method for imbalanced data, aiming to effectively identify minority class samples in an imbalanced data set through the cooperation mechanism of multiple agents, and improve the accuracy of the classification model when dealing with imbalanced data.

[0005] To achieve the above purpose, the present invention provides a multi-agent reinforcement learning classification method for imbalanced data, including the following steps:

[0006] Step 1: Establish a multi-agent Markov decision model;

[0007] Step 2: Set different rewards for the majority class and the minority class;

[0008] Step 3: According to the experience samples obtained from the interaction between the agent and the environment, preferentially sample important samples in the experience replay;

[0009] Step 4: Use a dual deep neural network to train the experience samples;

[0010] Step 5: Combine the centralized training of the agent with decentralized execution for distributed computing.

[0011] Optionally, during the process of step 1 modeling, the input imbalanced data is used as the state. The agent makes a classification action based on the state, and the classification action is passed to the environment. The environment gives a reward value according to the agent's classification action and passes the next state.

[0012] Optionally, in step 2, different reward values are set for the majority class and the minority class for imbalanced data. The reward values are set as follows:

[0013]

[0014] where a t is the classification action executed by the agent, s t is the sample state, l t is the class label, D P and D N belong to the minority class and the majority class samples respectively. The value of μ will be adjusted according to different dataset types: for image datasets, the reward value is the imbalance rate; for text datasets, the reward value is 0.1 times the imbalance rate. The imbalance rate is calculated by the ratio of the number of minority class samples to the number of majority class samples. The imbalance rate is calculated by the ratio of the number of minority class samples to the number of majority class samples.

[0015] Optionally, the execution process of step 3 includes the following steps:

[0016] Step 3.1: According to the empirical samples obtained from the interaction between the agent and the imbalanced data environment, prioritize the sampling of the experiences with high importance, and measure the importance through the temporal difference error (TD-error). For the i-th empirical sample, the temporal difference error formula is as follows:

[0017]

[0018] TD-error (i) represents the temporal difference error of the i-th experience, r is the reward, γ is the discount factor, a value between 0 and 1, indicating the agent's attention to future rewards; the higher the γ value, the more the agent values future rewards, and the lower the γ value, the more the agent focuses on current rewards; Q online is the current Q-network, used to calculate the Q value of taking action a i under the current state s i and the optimal action a i+1 is selected, Q target is the target Q-network, used to calculate the Q value in the next state s i+1The Q-value after selecting the optimal action below can effectively avoid the instability problem caused by the overestimation of the Q-value during the update process;

[0019] Step 3.2: Calculate the importance according to the temporal difference error. The importance formula is as follows:

[0020] Priority (i) = |TD-error (i) | + ρ

[0021] To avoid the importance being 0, a constant ρ is added;

[0022] Step 3.3: Calculate the sampling probability according to the importance. The expression is as follows:

[0023]

[0024] where N is the number of empirical samples and α is a hyperparameter;

[0025] Step 3.4: Introduce importance weights to adjust the bias brought by the experiences sampled with high importance, ensuring that during the use of importance sampling, the training process will not produce unfair learning effects due to some experiences being frequently selected. The importance weight formula is as follows:

[0026]

[0027] where β is a hyperparameter for adjusting the weights.

[0028] Optionally, two Q networks are introduced in Step 4 to avoid overestimation of the Q-value and approximate the cumulative reward by continuously adjusting the Q network. The execution process includes the following steps:

[0029] Step 4.1: Select an action;

[0030] In the current state, select an action using the current network. The expression is as follows:

[0031]

[0032] where θ is the parameter of the current network;

[0033] Step 4.2: Calculate the target Q-value;

[0034] First, select the optimal action for the next state using the current network and calculate the Q-value of this action using the target network. The target Q-value calculation formula is as follows:

[0035] y t = r t + γQ target (s t+1 ,a t+1 ; θ- )

[0036] Among them, θ - is the parameter of the target network;

[0037] Step 4.3: Update the Q network;

[0038] The update process is carried out by minimizing the loss function. Every once in a while or after a certain number of steps, the parameter θ - of the target network will be updated to the parameter θ of the online network.

[0039] Optionally, the centralized training process in Step 5 depends on a shared experience replay buffer, and the experiences of all agents will be collected in the shared experience replay buffer; this centralized experience replay pool can ensure that each agent shares experiences from different environments during the training process, thereby optimizing the classification strategy;

[0040] In the decentralized execution stage, each agent makes decisions independently based on its own local observations and decision-making models. During the execution process, it no longer depends on the shared experience buffer, but makes classification actions completely based on its own information, enabling each agent to perform classification actions in a decentralized manner according to the classification strategy learned in the centralized manner.

[0041] Combined with distributed computing, multiple heterogeneous servers are responsible for different tasks in the system, such as storing and processing data in the experience replay pool, training models, and executing the tasks of agents.

[0042] The present invention provides a multi-agent reinforcement learning classification method for imbalanced data. First, a Markov decision process is introduced to construct a classification model based on multi-agent reinforcement learning. Each agent observes the sample state and gives a corresponding classification result, and the environment provides feedback to the agent according to the classification result, including rewards or punishments, so as to strengthen the agent's learning of imbalanced samples. Secondly, different rewards are set for majority-class samples and minority-class samples respectively, and a dual deep neural network is used to calculate the cumulative reward. The training process is optimized through prioritized experience replay to improve the learning of minority classes. Finally, the centralized training and decentralized execution of agents are combined with distributed computing to optimize the interaction and learning efficiency of agents. Compared with most existing imbalanced data classification methods based on the data level and the algorithm level, the present invention can effectively improve the classification accuracy of minority classes and shorten the multi-agent collaborative classification time. Description of the Drawings

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic diagram of the step flow of a multi-agent reinforcement learning classification method for imbalanced data of the present invention.

[0045] Figure 2 It is a schematic diagram of the multi-agent Markov decision model structure of a multi-agent reinforcement learning classification method for imbalanced data of the present invention.

[0046] Figure 3 It is a schematic diagram of the F-measure results of a specific embodiment of the present invention on the IMDB dataset compared with other baseline methods.

[0047] Figure 4 It is a schematic diagram of the G-mean results of a specific embodiment of the present invention on the IMDB dataset compared with other baseline methods. Detailed implementation manners

[0048] The following details the embodiments of the present invention. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present invention, but should not be construed as a limitation to the present invention.

[0049] The present invention provides a multi-agent reinforcement learning classification method for imbalanced data, including the following steps:

[0050] Step 1: Establish a multi-agent Markov decision model;

[0051] Step 2: Set different rewards for the majority class and the minority class;

[0052] Step 3: According to the experience samples obtained from the interaction between the agent and the environment, preferentially sample the samples with high importance in the experience replay;

[0053] Step 4: Use a dual deep neural network to train the experience samples;

[0054] Step 5: Combine the centralized training and decentralized execution of the agent for distributed computing.

[0055] The execution process is as Figure 1 shown. The following further explains with specific implementation steps:

[0056] Step 1: Establish a multi-agent Markov decision model, as Figure 2 shown below:

[0057] The model consists of the following parts:

[0058] State space: It refers to the set of all possible states of the environment, and the attributes included in each state come from the samples in the dataset;

[0059] Dataset introduction:

[0060] (1) IMDB: This dataset is used for sentiment analysis, including movie reviews. Negative reviews are regarded as the positive class (minority class), and positive reviews are regarded as the negative class (majority class);

[0061] (2) Cifar-10: This dataset contains color images of 32×32 pixels. Label 1 is regarded as the positive class, and labels 3, 4, 5, and 6 are regarded as the negative class;

[0062] (3) Fashion-Mnist: This dataset contains color images of 28×2 pixels. Labels 0 and 2 are regarded as the positive class, and labels 1 and 3 are regarded as the negative class;

[0063] (4) Mnist: This dataset is used for digital image classification. Label 2 is regarded as the positive class, and the remaining digits are regarded as the negative class.

[0064] Action space: It refers to the set of all possible classification actions that the agent can choose in a given state. Each action represents a decision that the agent can take to affect the change of the environmental state.

[0065] Transition probability: It defines the probability that the environment transfers to the next state after the agent takes an action given the current state and action.

[0066] Reward function: The reward function defines the immediate reward obtained by the agent after taking an action in a certain state. The reward is usually a numerical value used to represent the goodness or badness of an action. In the context of imbalanced data, the reward function will set different rewards according to different classes (majority class or minority class).

[0067] Step 2: Set different rewards for the majority class and the minority class;

[0068] Specifically: In an imbalanced dataset, minority class samples are usually more important than majority class samples. To make the reinforcement learning model pay more attention to minority class samples, a reasonable reward function needs to be set so that the reward value of minority class samples is significantly higher than that of majority class samples. This can guide the model to pay more attention to minority class samples during the training process, thereby improving the classification performance of the model on minority class samples.

[0069] First, the reward value is set between zero and one. For minority class samples, a higher reward of 1 is set. If the classification is correct, a positive reward of 1 is obtained; otherwise, a negative reward of -1 is obtained.

[0070] Secondly, for majority class samples, the reward is set to a smaller value. For image datasets, the reward is set to the imbalance rate, while for text datasets, it is set to 0.1 times the imbalance rate.

[0071] Step 3: According to the experience samples obtained from the interaction between the agent and the environment, prioritize experience replay to sample samples with higher importance;

[0072] Step 3 includes the following processes:

[0073] First, calculate the temporal difference error of each experience sample, which is an index for evaluating the importance of the experience sample. This error reflects the difference between the currently estimated Q value and the target Q value;

[0074] Secondly, assign importance to each experience sample according to the temporal difference error. The larger the temporal difference error, the higher the importance of the experience sample, indicating that this sample is more important for training. The importance is the absolute value of the temporal difference error plus a non-zero constant to avoid zero importance;

[0075] Then, calculate the sampling probability according to the importance. Sample experience samples with high importance according to the sampling probability. Normalize the importance to obtain the sampling probability and adjust the weight through exponentiation;

[0076] Finally, to avoid frequent sampling of high-importance samples, it is necessary to adjust the weights of the sampled samples to ensure fairness in the contribution of each sample during the training process through importance sampling weights.

[0077] Step 4: Use a double deep neural network to train the experience samples.

[0078] Specifically: First, initialize two Q networks: the current network and the target network. Randomly sample a batch of experience samples from the experience replay buffer. Each experience sample includes the current state, the executed action, the obtained reward, and the next state;

[0079] Secondly, use the current network to estimate the Q value of each action in the current state, and select the Q value corresponding to the actually executed action as the current Q value;

[0080] Immediately afterwards, use the current network to estimate the Q value of each action in the next state, select the action with the largest Q value, and use the target network to estimate the Q value of the selected action in the next state as the target Q value;

[0081] Then, calculate the error between the current Q-value and the target Q-value, and use the Huber loss function to avoid over-punishing outliers. The Huber loss combines the advantages of the mean squared error and the absolute error. It behaves like the mean squared error when the error is small, enabling more accurate data fitting; when the error is large, it behaves like the absolute error, reducing over-punishment of outliers.

[0082] Finally, use the Adam optimizer to update the parameters of the current network to minimize the loss function. The parameters of the current network are updated through gradient descent, and the parameters of the current network are copied to the target network regularly.

[0083] Step 5: Combine centralized training and decentralized execution of the agent with distributed computing.

[0084] The following is illustrated by comparing with the remaining baseline methods in specific embodiments:

[0085] First, distribute the agents to multiple heterogeneous servers. The information of the heterogeneous servers is as follows:

[0086] Table 1 Comparison table of duration results between the present invention and the remaining baseline methods

[0087]

[0088] Secondly, collect the experiences of the agents into a shared experience replay buffer, and use the experience samples in the shared experience pool to train the centralized training model;

[0089] In the decentralized execution stage, each agent makes an independent classification decision according to its own state and environment during the execution stage.

[0090] Please refer to Figure 3 and Figure 4 , in the specific embodiment of the present invention, it is compared with the existing baseline methods (DQNimb, MADQN, DNN, GAN, and RUS) through different data sets. The following is a schematic diagram of the comparison results of the IMDB data set with other baseline methods:

[0091] Figure 3 It is a schematic diagram of the F-measure result of the specific embodiment of the present invention in the IMDB data set compared with other baseline methods.

[0092] Figure 4 It is a schematic diagram of the G-mean result of the specific embodiment of the present invention in the IMDB data set compared with other baseline methods.

[0093] Table 2 shows the duration results of a multi-agent reinforcement learning classification method for imbalanced data in the present invention compared with the baseline methods MADQN and DQNimb on the IMDB, Cifar-10, Fashion-Mnist, and Mnist datasets.

[0094] Table 2 Comparison table of duration (in seconds) results between the present invention and other baseline methods

[0095]

[0096] Therefore, the chart shows that this method has a relatively significant improvement in F-measure and G-mean under different datasets. Compared with the single-agent reinforcement learning algorithm, the time increase is not significant. For the multi-agent reinforcement learning algorithm, this method shows an obvious improvement in speed.

[0097] In summary, the present invention improves the classification accuracy and effectively reduces the time overhead brought by multi-agents. Compared with most existing imbalanced data classification methods based on the data level and the algorithm level, the present invention can effectively improve the classification accuracy of the minority class and enhance the multi-agent cooperation efficiency.

[0098] The above-disclosed are only one or more preferred embodiments of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A multi-agent reinforcement learning classification method for imbalanced data, characterized in that: The following steps are involved: Step 1: Establish a multi-agent Markov decision model; Step 2: Set up different rewards for the majority class and the minority class; Step 3: Based on the experience samples obtained by the interaction between the agent and the environment, prioritize the samples with high experience replay sampling importance; Step 4: Use the dual deep neural network to train the experience samples; Step 5: Combine centralized agent training with decentralized execution for distributed computing.

2. The multi-agent reinforcement learning classification method for imbalanced data according to claim 1, characterized in that: In the process of modeling in step 1, the input unbalanced data is used as the state, and the agent makes classification actions according to the state. The classification actions are passed to the environment. The environment gives a reward value according to the classification action of the agent and passes the next state.

3. The multi-agent reinforcement learning classification method for imbalanced data as claimed in claim 2, characterized in that: In step 2, different reward values ​​are set for the majority class and the minority class for the unbalanced data. The reward values ​​are set as follows: Among them, a t is the classification action performed by the agent, s t is the sample state, l t is the category label, D P Follow D N The values ​​of μ are adjusted according to different dataset types: for image datasets, the reward value is the imbalance rate; for text datasets, the reward value is 0.1 times the imbalance rate, and the imbalance rate is calculated by the ratio of the number of minority class samples to the number of majority class samples.

4. The multi-agent reinforcement learning classification method for imbalanced data as claimed in claim 3, characterized in that: The execution process of step 3 includes the following steps: Step 3.1: Based on the experience samples obtained by the interaction between the agent and the unbalanced data environment, the experience with high importance is sampled first, and the importance is measured by the time difference error. For the i-th experience sample, the time difference error formula is as follows: TD-error (i) represents the temporal difference error of the i-th experience, r is the reward, γ is the discount factor, a value between 0 and 1, indicating the agent's attention to future rewards; a higher γ value means that the agent attaches more importance to future rewards, and a lower γ value means that the agent pays more attention to current rewards; Q online is the current Q network, used to calculate the current state s i Take action a i The Q value of the optimal action a i+1 , Q target is the target Q network, used to calculate the next state s i+1 The Q value after selecting the optimal action; Step 3.2: Calculate the importance based on the time series difference error. The importance formula is as follows: Priority (i) =|TD error (i) |+ρ To avoid the importance being 0, a constant ρ is added; Step 3.3: Calculate the sampling probability based on the importance, the expression is as follows: Among them, N is the number of empirical samples, and α is a hyperparameter; Step 3.4: Introduce importance weights to adjust the deviations caused by the experience of high importance sampling. The importance weight formula is as follows: Among them, β is a hyperparameter for adjusting weights.

5. The multi-agent reinforcement learning classification method for imbalanced data according to claim 4, characterized in that: In step 4, two Q networks are introduced to avoid overestimation of Q value and to approximate the cumulative reward by continuously adjusting the Q network. The execution process includes the following steps: Step 4.1: Select an action; In the current state, use the current network to select an action. The expression is as follows: Among them, θ is the parameter of the current network; Step 4.2: Calculate the target Q value; First, use the current network to select the optimal action for the next state, and use the target network to calculate the Q value of the action. The target Q value calculation formula is as follows: y t =r t +γQ target (s t+1 ,a t+1 ;θ - ) Among them, θ - are the parameters of the target network; Step 4.3: Update the Q network; The update process is carried out by minimizing the loss function. Every once in a while or a number of steps, the parameters θ of the target network are updated. - will be updated as the parameters θ of the online network.

6. The multi-agent reinforcement learning classification method for imbalanced data according to claim 5, characterized in that: Step 5: The centralized training process relies on a shared experience replay buffer, into which the experience of all agents is collected. In the decentralized execution phase, each agent makes independent decisions based on its own local observations and decision-making models. The execution process no longer relies on a shared experience buffer, but instead makes classification actions entirely based on its own information.

Images