A data detection method based on hierarchical three-stage deep reinforcement learning

Abstract

The application discloses a data detection method based on layered three-stage deep reinforcement learning, relates to the technical field of network intrusion detection, and comprises the following steps: acquiring traffic data samples, inputting the traffic samples into a binary classifier, obtaining a suspicious sample set based on classification judgment of a threshold value passed through the binary classifier, inputting suspicious attack samples into a multi-classifier, obtaining a probability vector of each category to which the suspicious attack samples belong by the multi-classifier, constructing a category confusion relationship graph based on the probability vector and classifying to obtain a confusion sample set in combination with the category confusion relationship graph, constructing a corresponding number of expert classifications with weights according to the confusion sample set, discriminating corresponding confusion sample subsets based on category weights by the expert classifications, and obtaining a final confusion result set. The application significantly improves the intrusion prediction performance for the minority class and the confusion class by using the above method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network intrusion detection, and particularly relates to a data detection method based on layered three-stage deep reinforcement learning. BACKGROUND

[0002] With the increasing digitization of modern society, the cyberspace is facing increasingly severe security threats. The popularity of intelligent and digital technologies has led to a surge in the number of Internet of Things devices, which not only creates more exploitable targets for attackers, but also makes a single intrusion potentially cause greater damage. This technological competition between attackers and defenders has created an urgent need for the development of more intelligent and automated network security mechanisms. As a key defense line for network security, Network Intrusion Detection Systems (NIDS) play an indispensable role in real-time monitoring and identifying malicious traffic.

[0003] In recent years, data-driven NIDS has become the mainstream of research in this field, especially methods based on machine learning and deep learning, which can automatically learn complex attack patterns from massive traffic data, effectively breaking through the limitations of traditional signature matching methods.

[0004] Despite the promising prospects, the performance of data-driven NIDS in practical applications is severely constrained by the inherent characteristics of real-world network data. The first challenge is the class imbalance problem. In real network environments, the number of normal traffic far exceeds that of malicious traffic, and some specific attack types are extremely rare. This severe skew in data distribution causes the model to tend to predict samples as normal traffic or samples of more attack categories, thus showing a low detection rate for important minority class attacks such as worms, backdoors, etc.

[0005] However, a more deep-seated and more destructive problem is the class confusion problem. In order to evade detection, attackers often deliberately forge attack traffic to make it highly similar to normal communication patterns at the feature level. This causes serious overlap in feature space between different classes (such as normal traffic and some attacks, or between different attacks), making it difficult for classifiers to draw clear decision boundaries.

[0006] Therefore, there is an urgent need for a data detection method based on layered three-stage deep reinforcement learning to improve the above problems. SUMMARY

[0007] To solve the above problems, the present application proposes a data detection method based on layered three-stage deep reinforcement learning, which aims to systematically address the two challenges of class imbalance and class confusion through the strategy of "divide and conquer", including the following steps:

[0008] S1, obtaining traffic data samples , the traffic sample inputting the LightGBM binary classifier , the model output vector , the LightGBM binary classifier is trained on all training data, where all attack classes are merged into a single "attack" class, with the label being the "normal" class, is the probability of the "attack" class, is the probability of the "normal" class.

[0009] by the threshold of the LightGBM binary classifier , if the probability of the attack is less than , then the traffic data sample is a benign sample, and is classified as ;

[0010] if the probability of the attack is greater than or equal to , then the traffic data sample is a suspicious sample set ;

[0011] In order to determine the threshold, the present application adopts a grid search strategy based on 5-fold cross-validation. Specifically, the threshold is determined in the training phase by a robust strategy combining cross-validation and grid search. First, 5-fold cross-validation is performed on the entire training set to generate an Out-of-Fold (OOF) prediction probability for each training sample. This OOF prediction array can be regarded as an unbiased estimate of the model's performance on unseen data. Subsequently, the optimal threshold is found on the OOF predictions of the entire training sample by grid search. That is, an exhaustive search is performed within a predetermined threshold range, and the threshold that maximizes the Macro F1-score is selected. Macro F1-score is a key indicator for evaluating the performance of imbalanced multi-classification tasks, which treats each class equally regardless of the number of samples.

[0012] S2, inputting the suspicious attack sample set into the LightGBM multi-classifier , the LightGBM multi-classifier obtains the predicted class set of the suspicious attack sample set ;

[0013] Based on the predicted class set of all samples , a class confusion relationship diagram is constructed, and the confusion sample set is classified by combining the class confusion relationship diagram .

[0014] wherein the predicted class set represents the set of classes that all samples in the second stage are predicted to belong to.

[0015] In the validation stage, if a sample is predicted to be an easy-to-classify sample, it is classified into the easy-to-classify sample set EATK, and if it is a confused sample, it is classified into the confused sample set Con.

[0016] In the test stage, only the out-of-fold prediction result set of each fold cross-validation is represented, and the out-of-fold prediction confusion matrix OOF of each fold is obtained by combining the true label of each sample.

[0017] S2, on the one hand, is a comprehensive multi-classification discrimination of these suspicious samples, so that most of the attack samples with obvious features and easy to distinguish are directly identified and the results are output, and on the other hand, it actively and quantitatively identifies those classes that show high confusion in the preliminary classification. Through this data-driven identification mechanism, the confused class sample set is provided for the expert model in the third stage, so as to get rid of the limitation of the traditional method which only relies on the heuristic rule of "minority class" to define difficult samples.

[0018] S3, according to the confused sample set construct a corresponding number of LightGBM expert classifications with weights , the LightGBM expert classification discriminates the corresponding confused sample subset based on the class weight to obtain the final confused result set ;

[0019] The class weight of the LightGBM expert classification is dynamically optimized by the DDQN agent.

[0020] The agent is in a DDQN environment (DDQN Env), and according to the state of the current model, the agent outputs an action to adjust the weight, and obtains a reward according to the evaluation result , so as to learn an optimal weight adjustment strategy. Finally, each LightGBM expert classifier applies the best weight obtained during training to the corresponding confused sample subset to obtain the final confused result set .

[0021] Preferably, based on the predicted class set of all samples , a class confusion relationship diagram is constructed and the confused sample set is classified in combination with the class confusion relationship diagram The specific content is:

[0022] In the training phase, in order to eliminate the influence of the number difference of various categories of samples and quantify the confusion degree between categories, through the confusion recognition mechanism based on 5-fold cross-validation, the entire suspicious sample set is obtained The confusion matrix on the validation set ;

[0023] Based on this confusion matrix, the confusion weight between each category is first calculated, and then the symmetric confusion weight between any two categories is obtained by adding the confusion weight between any two categories ;

[0024] The method of graph theory is introduced to identify the structured confusion relationship, and based on the confusion threshold , all category pairs with confusion weight higher than the threshold are identified, and a category confusion relationship graph is constructed;

[0025] In the reasoning phase, the suspicious sample set is distributed based on the reasoning of the category confusion relationship graph ;

[0026] If the suspicious sample belongs to an isolated node in the category confusion relationship graph, it is defined as an attack category that is easy to classify , and the result is directly output;

[0027] If the suspicious sample belongs to a connected subgraph connected by multiple confusion edges in the category confusion relationship graph, the sample is classified as a confusion sample and stored in the confusion sample set , each connected subgraph is regarded as a confusion category group, and the next step of confusion category classification is performed.

[0028] Preferably, the confusion recognition mechanism based on 5-fold cross-validation includes generating out-of-fold prediction OOF for each training sample through 5-fold cross-validation on all suspicious attack samples in order to obtain an unbiased estimate of the model generalization ability;

[0029] Based on the true label of the suspicious attack sample and the out-of-fold prediction OOF, a confusion matrix is constructed (in essence, the confusion matrix OOF of each fold is added to obtain the confusion matrix M of all training set samples in the validation process).

[0030] Preferably, the symmetric confusion weight between the category and the category is expressed as:

[0031] ;

[0032] ;

[0033] ;

[0034] wherein, is the proportion of samples that are misclassified as class is the number of samples in the confusion matrix with true label class i but misclassified as class j, is the number of samples in the confusion matrix with predicted label i but actual label j, i and j represent two different attack classes respectively.

[0035] Preferably, based on the confusion threshold , all class pairs with confusion weight higher than the threshold are identified, and the specific content of the class confusion relationship graph is:

[0036] The class confusion relationship graph is an undirected weighted graph , wherein, represents the set of all points of the confusion relationship graph, and each point represents an attack class; represents the set of edges between each point, that is, the confusion relationship between each point.

[0037] wherein each attack class is a vertex ;

[0038] If the confusion weight between any two classes and is higher than , an edge is added between class and class , and finally all confusion class groups are identified by finding all connected components of the confusion graph.

[0039] Preferably, the state space of the DDQN agent is , including state ;

[0040] wherein, ;

[0041] wherein, t is the current time step, represents the number of classes in the confusion group, represents the weight of class at the current time step , represents the class​​​​​ Macro F1 score achieved after training on the current training set, denotes the current time step Macro F1 score of all confusion classes.

[0042] The agent knows the performance of the current classifier through the current state.

[0043] Through the state space, the DDQN agent gets the influence of the weight of each class on the macro F1 score of each class and the overall macro F1 score.

[0044] Preferably, the action space is designed as a set of discrete operations that fine-tune the class weights, and the action space of the DDQN agent is ;

[0045] At the current time step , the DDQN agent selects an action , and the action space is defined as:

[0046] ;

[0047] wherein denotes the total number of actions, denotes the class to be adjusted, denotes the specific operation performed on the class at the current time step as action .

[0048] Preferably, the reward function of the DDQN agent is ;

[0049] The reward function is the key to guide the agent to learn, and the goal of the present application is to improve the overall balanced performance of the model, therefore, the reward is the increment of the macro F1 score, and the expression of the reward is:

[0050] ;

[0051] wherein is the macro F1 score obtained on the validation set after performing action , that is, at the current time step, is the macro F1 score obtained on the validation set at the last time step.

[0052] Since this reward directly reflects the recognition of the model for the confusion classes, through this reward, the agent can adjust the strategy in real time and strive to maximize the macro F1 score, so as to improve the recognition performance of the model in the case of class confusion.

[0053] Preferably, the DDQN agent adopts a Double Deep Q Network (DDQN) as the core algorithm.

[0054] The Double Deep Q Network comprises a current Q network for action selection, whose parameters are ;

[0055] a target network for action value evaluation, whose parameters are ;

[0056] Preferably, during the training process, the interactive experience of the DDQN agent, i.e. a series of state transition tuples , is stored in an experience replay pool ;

[0057] During the training, a small batch of data is randomly sampled from ;

[0058] For each sample , the DDQN updates the parameters of the current Q network by minimizing a mean squared error loss function :

[0059] ;

[0060] wherein is the value prediction of the current Q network for performing action in state ;

[0061] The expression of the TD target is:

[0062] ;

[0063] wherein is the immediate reward obtained after performing action , is a discount factor, a hyperparameter between , for weighing the importance of immediate reward and future reward, is the action with the highest value in the next state selected by the current Q network, and represents the part of using the target network to evaluate the value of the selected action .

[0064] By decoupling the two steps of action selection and value evaluation to two different networks, the DDQN effectively avoids the bootstrap overestimation problem caused by the same network being responsible for both selection and evaluation in the standard DQN.​

[0065] To sum up, compared with the traditional technology, the data detection method based on hierarchical three-stage deep reinforcement learning has the following advantages:

[0066] 1. Based on the efficient performance of gradient decision tree in table type data, LightGBM is selected as the base classifier. Compared with the computationally intensive deep learning model, LightGBM significantly reduces memory occupation and computing cost when processing large-scale structured traffic data. At the same time, it realizes accurate extraction and high-performance identification of traffic features while ensuring training efficiency by virtue of its leaf growth strategy.

[0067] 2. A coarse-to-fine cascade detection process is designed: the first stage uses LightGBM binary classifier to quickly filter normal traffic and lock malicious traffic; the second stage uses LightGBM multi-classifier to preliminarily divide the malicious traffic and locate the seriously overlapping confusion categories; the third stage deploys special classifiers for the identified confusion category clusters. This framework effectively alleviates the class imbalance and long-tail distribution problems commonly existing in network traffic by decoupling the task complexity layer by layer.

[0068] 3. A confusion category identification method based on cross-validation is designed. In the second stage of training, a 5-fold cross-validation strategy is used to obtain the out-of-bag prediction results of the full sample. By constructing a global cumulative confusion matrix, the random bias of single division is eliminated, ensuring the robustness of misclassification statistics. Based on this, the inter-class bidirectional confusion weight is further calculated to construct a symmetric category similarity matrix. The categories are mapped to graph nodes, and the bidirectional confusion weight is mapped to edge weight. By setting the confusion threshold, the graph structure is pruned and filtered. Finally, multiple independent high-confusion category clusters are identified by using connected component analysis, and a special fine-grained classifier is constructed for each cluster in the third stage to accurately process the confusion difficult-to-divide samples.

[0069] 4. The class weight of the third stage confusion category identification LightGBM model is adjusted by introducing the deep reinforcement learning algorithm DDQN. By introducing deep reinforcement learning, the agent can autonomously find the efficient weight combination that maximizes the minority class detection performance, thereby overcoming the dependence of traditional optimization methods on human-set search space. At the same time, the experience replay mechanism of the Off-policy strategy of DDQN is used to maximize the reuse of historical data for learning, avoiding frequent model evaluation and significantly reducing computational overhead.

[0070] The technical method of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1A data detection method based on hierarchical three-stage deep reinforcement learning;

[0072] Figure 2 A specific training process of the LightGBM expert classifier;

[0073] Figure 3 Algorithm steps of the DDQN-based confusion class expert classifier training algorithm;

[0074] Figure 4 DDQN network update process. DETAILED DESCRIPTION

[0075] The technical method of the present application is further described below by means of the accompanying drawings and examples. It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these examples do not limit the scope of the present application.

[0076] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the application or its applications or uses.

[0077] Techniques, systems, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0078] In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0079] Unless otherwise defined, technical and scientific terms used herein should have the meaning commonly understood by one of ordinary skill in the art to which this application belongs.

[0080] Example 1

[0081] A data detection method based on hierarchical three-stage deep reinforcement learning (H3-DRL) is composed of a hierarchical detection architecture consisting of three logical stages. Its data processing flow and decision logic are shown in Figure 1 .

[0082] An input network traffic sample will be processed according to the following process:

[0083] S1, obtain traffic data sample , traffic sample is input into the LightGBM binary classifier , the model output vector ;

[0084] threshold value of the LightGBM binary classifier determining that the probability of attack is less than the traffic data sample is a benign sample and is classified as ;

[0085] determining that the probability of attack is greater than or equal to the traffic data sample is a suspicious sample set ;

[0086] S2, inputting the suspicious attack sample set into a LightGBM multi-classifier, and obtaining a predicted class set of the suspicious attack sample set by the LightGBM multi-classifier ; ;

[0087] constructing a class confusion relationship graph based on the probability vector and classifying to obtain a confusion sample set in combination with the class confusion relationship graph ;

[0088] S3, constructing a corresponding number of LightGBM expert classifications with weights according to the confusion sample set , the LightGBM expert classification is based on class weights to distinguish the corresponding confusion sample subset, and obtains a final confusion result set ;

[0089] The class weights of the LightGBM expert classification are dynamically optimized by a DDQN agent.

[0090] The specific training process of the LightGBM expert classification is shown in Figure 2 .

[0091] The DDQN agent adopts a double deep Q network (DDQN) as a core algorithm, and detailed algorithm steps of the process are given in Figure 3 , which is a DDQN network update process. Figure 4

[0092] Embodiment 2

[0093] In order to comprehensively evaluate the performance and generalization ability of a data detection method based on a hierarchical three-stage deep reinforcement learning in different scenarios, two benchmark data sets widely recognized in the field of intrusion detection, UNSW-NB15 and NSL-KDD, are selected in this embodiment, and the original UNSW-NB15 and NSL-KDD data sets are subjected to a preprocessing process of data cleaning, feature type separation and coding, and feature selection based on a random forest.​​​​​​​​​​

[0094] This embodiment conducts a comprehensive performance comparison between H3-DRL and a series of mainstream imbalanced learning single-stage models based on LightGBM. The experimental results are shown in Table 1.

[0095] Table 1 Performance comparison between H3-DRL and mainstream imbalanced processing methods on the UNSW-NB15 test set

[0096]

[0097] From the experimental results in Table 1, the effectiveness of different imbalanced processing strategies can be analyzed in depth. First, the standard LightGBM model achieves the highest accuracy, but its macro-average recall is the worst among all comparison models. This phenomenon clearly reveals the misleading nature of accuracy in imbalanced scenarios: the model achieves high accuracy by accurately predicting the majority class with a large sample size, but at the cost of sacrificing the detection ability of the minority class, resulting in poor overall balanced performance.

[0098] Second, the present application observes the trade-offs and limitations of traditional SMOTE and cost-sensitive learning methods in improving performance. Compared with the standard LGBM, these two methods have significantly improved macro-average recall, indicating that the model has indeed been guided to more actively identify minority class samples. However, this improvement in recall comes at the expense of a huge sacrifice in precision, with both methods having a macro-average precision of less than 58%. This shows that, whether through data interpolation or static weight allocation, the model has found more minority class samples while producing a large number of false positives. This aggressive strategy of "better a thousand times wrong than once right" ultimately results in their performance on the comprehensive performance indicator Macro-F1-score being even worse than the standard LGBM, proving that it is not an ideal solution.

[0099] Finally, the H3-DRL framework proposed in this application stands out among all the models, demonstrating its excellent ability to achieve the best balance between precision and recall. It not only achieved the highest macro-average recall (77.36%), but more importantly, it also maintained the highest macro-average precision (69.64%) in the entire field. This result strongly proves that the "divide and conquer + dynamic optimization" strategy of this application is much more sophisticated than simple data sampling or static weighting. By introducing DDQN in the third stage for intelligent optimization, the framework of this application finds a best balance point between "precision" and "recall" for highly confused classes. It is this excellent balance ability that makes it surpass all single-stage baseline models in the core indicator of macro-average F1 score, reaching 0.7150, which is a comprehensive performance. This significant advantage in macro-average indicators directly proves that the H3-DRL framework is more effective than traditional data-level or algorithm-level methods in alleviating the performance deviation problem caused by class imbalance.

[0100] Embodiment 3

[0101] To prove the effectiveness of the three-stage architecture and the DDQN optimization strategy in dealing with class imbalance and class confusion, this application designs and conducts a series of ablation experiments. This application creates two variant models by removing or replacing key modules of the framework: single-stage DDQN-LGBM and three-stage Bayesian-LGBM. All models are evaluated on the same preprocessed dataset, with the core comparison indicator being the macro-average F1 score (Macro-F1 score), and the results are shown in Table 2, with the attack classes being the original class names of the dataset.

[0102] Table 2 Comparison of macro-average F1 scores of each model in the ablation experiment on each attack class of UNSW-NB15

[0103]

[0104] Note: To verify the model's recognition efficiency for attack traffic, this application excludes the Normal class and adds the standard LGBM as the final reference benchmark in the table.

[0105] Through the three sets of comparative experiments, this application can draw the following conclusions:

[0106] First, by comparing the standard LGBM with the single-stage DDQN-LGBM, this application finds that using DDQN to adjust the weights can improve the classification efficiency of the classifier. And like Backdoor, Worms, and other minority macro-average F1 scores in the training set, they also get a considerable improvement after the weight adjustment of DDQN. This proves that the weight adjustment method based on deep reinforcement learning is feasible.

[0107] Second, by comparing the single-stage DDQN-LGBM with the complete H3-DRL framework, it can be seen that the introduction of the three-stage architecture brings an additional performance gain of 4.32%. And the Analysis category completely ignored in the single-stage DDQN-LGBM can be identified in H3-DRL, which proves that structurally separating the detection process can effectively alleviate the identification problem of class imbalance.

[0108] Third, by comparing the results of the three-stage Bayesian-LGBM and H3-DRL, except for the non-confusion category (because the first two stages are exactly the same, so the results of the non-confusion category are also exactly the same), H3-DRL has a significant improvement in the identification efficiency of the confusion category. Especially the Analysis category and the Dos category which have strong confusion with the Exploits category, which are difficult to identify by other classifiers, H3-DRL performs best. Finally, the improvement of the identification efficiency of the confusion category is reflected in the overall macro-average F1 score, reaching 0.7150. Thus it can be proved that the confusion category identification method of the second stage can effectively find out the confusion category, thereby alleviating the class confusion problem caused by the inherent similarity of features.

[0109] In summary, the present application proposes a joint framework combining structured task decomposition and intelligent strategy learning, which significantly improves the performance of intrusion prediction, especially for minority classes and confusion classes. The model proposed by the present application utilizes the closed-loop interaction between the agent and the classification environment. The experimental results obtained provide detailed performance analysis on the UNSW-NB15 and NSL-KDD datasets and comparative analysis of H3-DRL with various baselines. Finally, the comparison and evaluation results show that the proposed H3-DRL is superior to other state-of-the-art methods and traditional imbalance processing techniques such as SMOTE and cost-sensitive learning in terms of prediction performance, especially in terms of macro-average F1 score.

[0110] Finally, it should be noted that: the above embodiments are only used to illustrate the technical method of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can still be modified or replaced by equivalents, and these modifications or equivalent replacements cannot make the modified technical method deviate from the spirit and scope of the technical method of the present application.

Claims

1. A data detection method based on hierarchical three-stage deep reinforcement learning, characterized in that, Includes the following steps: S1. Obtain traffic data samples Traffic samples Input LightGBM binary classifier Model output vector , This is a normal probability. Threshold of LightGBM binary classifier Determine the probability of an attack. Less than Then the traffic data sample This is a benign sample, categorized as ; If the probability of attack Greater than or equal to Then the traffic data sample For suspicious sample set ; S2, Transfer suspicious attack samples Input LightGBM multi-classifier The LightGBM multi-classifier obtained a set of suspected attack samples. Predicted category set ; Predicted category set based on all samples Construct a category confusion graph and classify the confused samples based on the category confusion graph. ; S3, Based on the confused sample set Construct a corresponding number of weighted LightGBM expert classifications LightGBM expert classification uses class weights to distinguish the corresponding confused sample subsets, thus obtaining the final confused result set. ; The category weights of the LightGBM expert classification Dynamically optimized by DDQN agents; Predict the category based on each sample. Construct a category confusion graph and classify the confused samples based on the category confusion graph. The specific content is as follows: The predicted class of each sample is obtained by obfuscating it using an obfuscation recognition mechanism based on 5-fold cross-validation. This leads to the collection of suspicious samples. Confusion matrix on the validation set ; Based on this confusion matrix, the symmetric confusion weights between any two categories can be calculated. ; Based on confusion threshold Identify all category pairs with confusion weights higher than the threshold and construct a category confusion graph; Using category confusion relation graph reasoning to segment suspicious sample sets ; If a suspicious sample is an isolated node in the category confusion graph, it is defined as an attack category that is easily classified. Output the result directly; If a suspicious sample belongs to a connected subgraph consisting of multiple confusing edges in the category confusion graph, then the sample is classified as a confused sample and stored in the confused sample set. middle; The confusion detection mechanism based on 5-fold cross-validation involves generating an out-of-fold prediction result set, i.e., the predicted class of the sample, for each training sample through 5-fold cross-validation on suspicious attack samples. ; An out-of-fold prediction confusion matrix (OOF) is constructed based on the out-of-fold prediction result set and the true labels of the suspected attack samples. A confusion matrix is ​​constructed based on the true labels of the suspected attack samples and all out-of-place prediction confusion matrices (OOF). ; Symmetric Confusion Weights For category and categories Symmetrical confusion weights between them; Based on confusion threshold The specific steps for identifying all category pairs with confusion weights higher than the threshold and constructing a category confusion graph are as follows: The category confusion graph is an undirected weighted graph. ,in, This represents the set of all points in the obfuscated graph, where each point represents an attack category. This represents the set of edges between each point, which is the confusion relationship between each point; Each attack category is a vertex. ; If any two categories and Confusion weights between Higher than In category and categories Add an edge between ; The state space of the DDQN agent is: , including status ; in, ; in, t For the current time step, Indicates the number of categories in the confusion group. Indicates category At the current time step The weight, Indicates the category under the current weight. The macro-average F1 score obtained after training on the current training set. This indicates the current time step. Macro average F1 score for all confusion categories; Through the state space, the DDQN agent obtains the influence of the weight of each category on the macro-average F1 score of each category and the overall macro-average F1 score; The action space of the DDQN agent is ; At the current time step The DDQN agent selects an action. Action space Defined as: ； Where m represents the number of actions. Indicates the category to be adjusted. This indicates that the specific operation performed on this class at the current time step is an action. ; The reward function of the DDQN agent is: ; award The bonus is the increase in the macro average F1 score. The expression is: ； in, It is to perform an action Then, the macro-average F1 score obtained on the validation set at the current time step. The macro-average F1 score obtained on the validation set at the previous time step; The DDQN agent uses a dual deep Q network as its core algorithm. A dual-depth Q-network consists of: a current Q-network used for action selection, whose parameters are... ; The target network used to evaluate the value of actions has the following parameters: ; During training, the interactive experience of the DDQN agent, namely a series of state transition tuples. It is stored in an experience replay pool. middle; During training, from A small batch of data is randomly sampled from the data. For each sample DDQN minimizes a mean squared error loss function. To update the parameters of the current Q network : ； in, Is the current Q network pair in state? Next action Value prediction; TD Target The expression is: ； in, It is to perform an action The instant reward obtained afterward It is a discount factor, one between The hyperparameters between these parameters are used to weigh the importance of immediate rewards versus future rewards. The current Q network selects the next state. The most valuable action ,and This section indicates that the target network is used to evaluate the selected action. The value of.

2. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 1, characterized in that, Symmetric Confusion Weights The calculation formula is: ； ； ； in, For category Misclassified as a category proportion, For category Misclassified as a category proportion, In the confusion matrix In the above, the number of samples whose true label is category i but were misclassified as category j is: In the confusion matrix In this context, the number of samples with predicted label i but actual label j is denoted as i, where i and j represent two different attack categories.

Images