Data detection method based on layered three-stage deep reinforcement learning

Abstract

The invention discloses a data detection method based on hierarchical three-stage deep reinforcement learning, which relates to the technical field of network intrusion detection, and comprises the following steps: acquiring a traffic data sample, inputting the traffic sample into a binary classifier, and performing classification judgment based on a threshold value of the binary classifier to obtain a suspicious sample set; the suspicious attack samples are input into a multi-classifier, the multi-classifier obtains a probability vector of the suspicious attack samples belonging to each category, a category confusion relation graph is constructed based on the probability vectors, a confusion sample set is obtained through classification in combination with the category confusion relation graph, and a corresponding number of expert classifications with weights are constructed according to the confusion sample set; and the expert classification is used for judging the corresponding confusion sample subsets based on category weights to obtain a final confusion result set. According to the method, the intrusion prediction performance is remarkably improved for minority classes and confusion classes.

Description

Technical Field

[0001] This invention relates to the field of network intrusion detection technology, and in particular to a data detection method based on hierarchical three-stage deep reinforcement learning. Background Technology

[0002] As modern society becomes increasingly digital, cyberspace faces ever-growing security threats. The proliferation of smart and digital technologies has led to a surge in the number of IoT devices, creating more targets for attackers and increasing the potential for larger-scale damage from a single intrusion. This technological battle between attackers and defenders urgently necessitates the development of more intelligent and automated cybersecurity mechanisms. As a critical line of defense for cybersecurity, Network Intrusion Detection Systems (NIDS) play an indispensable role in real-time monitoring and identification of malicious traffic.

[0003] In recent years, data-driven NIDS has become the mainstream of research in this field, especially methods based on machine learning and deep learning. These methods can automatically learn complex attack patterns from massive traffic data, effectively overcoming the limitations of traditional signature matching methods.

[0004] Despite its promising future, the performance of data-driven NIDS in practical applications is severely constrained by the inherent characteristics of real-world network data. The primary challenge is class imbalance. In real-world network environments, normal traffic far outweighs malicious traffic, and certain attack types are extremely rare. This severe skew in data distribution causes models to tend to predict samples as normal traffic or as the more prevalent attack category, resulting in lower detection rates for important minority attacks such as worms and backdoors.

[0005] However, a deeper and more destructive problem is class confusion. To evade detection, attackers often deliberately forge attack traffic to make it highly similar to normal communication patterns at the feature level. This leads to severe overlap between different categories (such as normal traffic and certain attacks, or between different attacks) in the feature space, making it extremely difficult for classifiers to define clear decision boundaries.

[0006] Therefore, there is an urgent need for a data detection method based on hierarchical three-stage deep reinforcement learning to improve the above problems. Summary of the Invention

[0007] To address the aforementioned issues, this application proposes a data detection method based on hierarchical three-stage deep reinforcement learning. This method aims to systematically address the two major challenges of class imbalance and class confusion through a "divide and conquer" strategy, including the following steps: S1. Obtain traffic data samples Traffic samples Input LightGBM binary classifier Model output vector The LightGBM binary classifier is trained on all training data, where all attack categories are merged into a single "attack" class, along with the "normal" class. For the probability of "attack", The probability is for the "normal" class.

[0008] Threshold of LightGBM binary classifier Determine the probability of an attack. Less than Then the traffic data sample This is a benign sample, categorized as ; If the probability of attack Greater than or equal to Then the traffic data sample For suspicious sample set ; To determine this threshold, this application employs a grid search strategy based on 5-fold cross-validation. Specifically, the threshold is determined during the training phase using a robust strategy combining cross-validation and grid search. First, 5-fold cross-validation is performed across the entire training set, generating an out-of-fold (OOF) prediction probability for each training sample. This OOF prediction array can be viewed as an unbiased estimate of the model's performance on unseen data. Subsequently, a grid search is used to find the optimal threshold based on this OOF prediction for all training samples. That is, an exhaustive search is performed within a preset threshold range, and the threshold that maximizes the macro average F1 score is selected. The macro F1 score is a key metric for evaluating the performance of imbalanced multi-class classification tasks, treating each class equally regardless of its sample size.

[0009] S2, Set up suspicious attack samples Input LightGBM multi-classifier The LightGBM multi-classifier obtained a set of suspected attack samples. Predicted category set ; Predicted category set based on all samples Construct a category confusion graph and classify the confused samples based on the category confusion graph. .

[0010] Among them, the prediction category set This represents the set of predicted categories for all samples in the second stage.

[0011] During the validation phase, if a sample is predicted to be an easily classifiable sample, it is classified into the easily classifiable sample set EATK; if it is a confusing sample, it is classified into the confusing sample set Con.

[0012] During the testing phase, only the out-of-fold prediction result set of each fold cross-validation is represented. Combined with the true label judgment of each sample, the out-of-fold prediction confusion matrix (OOF) of each fold is obtained.

[0013] S2 performs a comprehensive multi-class classification of these suspicious samples, directly identifying and outputting results for most of the attack samples with obvious and easily distinguishable features. Simultaneously, it proactively and quantitatively identifies categories that exhibit high levels of confusion in the initial classification. This data-driven identification mechanism provides a set of confused category samples for the third-stage expert model, thus overcoming the limitations of traditional methods that rely solely on the heuristic rule of "minority class" to define difficult samples.

[0014] S3, Based on the confused sample set Construct a corresponding number of weighted LightGBM expert classifications LightGBM expert classification uses class weights to distinguish the corresponding confused sample subsets, thus obtaining the final confused result set. ; The category weights of the LightGBM expert classification Dynamically optimized by DDQN agents.

[0015] An agent in a DDQN environment (DDQN Env) determines the current state of the model based on the current state of the agent. Output Action To adjust the weights and award rewards based on the evaluation results. This allows each LightGBM expert classifier to learn an optimal weight adjustment strategy. Finally, each classifier applies its best weights, obtained during training, to discriminate against the corresponding subset of confused samples, resulting in the final confused result set. .

[0016] Preferably, a predicted category set based on all samples. Construct a category confusion graph and classify the confused samples based on the category confusion graph. The specific content is as follows: In the training phase, to eliminate the impact of differences in the number of samples in each category and to quantify the degree of confusion between categories, a confusion identification mechanism based on 5-fold cross-validation is used to obtain the entire suspicious sample set. Confusion matrix on the validation set ; Based on this confusion matrix, the confusion weight between each category is first calculated, and then the confusion weights between any two categories are added together to obtain the symmetric confusion weight between any two categories. ; Graph theory methods are introduced to identify structured confusion relations, based on a confusion threshold. All category pairs with confusion weights higher than the threshold are identified, and a category confusion relationship graph is constructed. In the reasoning stage, suspicious sample sets are segmented based on category confusion relationship graph reasoning. ; If a suspicious sample is an isolated node in the category confusion graph, it is defined as an attack category that is easily classified. Output the result directly; If a suspicious sample belongs to a connected subgraph consisting of multiple confusing edges in the category confusion graph, then the sample is classified as a confused sample and stored in the confused sample set. In this process, each connected subgraph is treated as a group of confusion categories, and the next step of confusion category classification is performed.

[0017] Preferably, the confusion detection mechanism based on 5-fold cross-validation includes generating an out-of-fold prediction (OOF) for each training sample by performing 5-fold cross-validation on all suspected attack samples in order to obtain an unbiased estimate of the model's generalization ability; Construct a confusion matrix based on the true labels and out-of-flight predictions (OOF) of the suspected attack samples. (Essentially, it involves summing the confusion matrices OOF of each fold to obtain the confusion matrix M of all training set samples during the validation process.)

[0018] Preferred, symmetric confusion weights For category and categories The symmetric obfuscation weights between them are expressed as follows: ; ; ; in, For category Misclassified as a category proportion, For category Misclassified as a category proportion, In the confusion matrix In the above, the number of samples whose true label is category i but were misclassified as category j is: In the confusion matrix In this context, the number of samples with predicted label i but actual label j is denoted as i, where i and j represent two different attack categories.

[0019] Preferably, based on the confusion threshold The specific steps for identifying all category pairs with confusion weights higher than the threshold and constructing a category confusion graph are as follows: The category confusion graph is an undirected weighted graph. ,in, This represents the set of all points in the obfuscated relationship graph, where each point represents an attack category. This represents the set of edges between each point, which is the confusion relationship between each point.

[0020] Each attack category is a vertex. ; If any two categories and Confusion weights between Higher than In the category and categories Add an edge between Finally, all confusion category groups are identified by finding all connected components of the confusion graph.

[0021] Preferably, the state space of the DDQN agent is: , including status ; in, ; in, t For the current time step, Indicates the number of categories in the confusion group. Indicate category At the current time step The weight, Indicates the category under the current weight. The macro-average F1 score obtained after training on the current training set. This indicates the current time step. The macro average F1 score for all confusion categories.

[0022] The agent can determine the performance of the current classifier based on the current state.

[0023] Through the state space, the DDQN agent obtains the influence of the weight of each category on the macro-average F1 score of each category and the overall macro-average F1 score.

[0024] Preferably, the action space is designed as a set of discrete operations that fine-tune the class weights, and the action space of the DDQN agent is... ; At the current time step The DDQN agent selects an action. Action space Defined as: ; in, Indicates the total number of actions. Indicates the category to be adjusted. This indicates that the specific operation performed on this class at the current time step is an action. .

[0025] Preferably, the reward function of the DDQN agent is: ; The reward function is crucial for guiding the agent's learning. This application aims to improve the overall balance performance of the model; therefore, the reward... The bonus is the increase in the macro average F1 score. The expression is: ; in, It is to perform an action Then, that is, the macro-average F1 score obtained on the validation set at the current time step. It is the macro-average F1 score obtained on the validation set at the previous time step.

[0026] Since this reward directly reflects the model's ability to identify confused categories, the agent can adjust its strategy in real time to maximize the macro-average F1 score, thereby improving the model's recognition performance in cases of category confusion.

[0027] Preferably, the DDQN agent uses Dual Deep Q Network (DDQN) as its core algorithm; The dual-depth Q-network consists of: a current Q-network used for action selection, whose parameters are... ; The target network used to evaluate the value of actions has the following parameters: ; Preferably, during training, the interaction experience of the DDQN agent, i.e., a series of state transition tuples, is used. It is stored in an experience replay pool. middle; During training, from A small batch of data is randomly sampled from the data. For each sample DDQN minimizes a mean squared error loss function. To update the parameters of the current Q network : ; in, Is the current Q-network pair in state? Next action Value prediction; TD Target The expression is: ; in, It is to perform an action The instant reward obtained afterward It is a discount factor, one between The hyperparameters between these parameters are used to weigh the importance of immediate rewards versus future rewards. The current Q network selects the next state. The most valuable action ,and This section indicates that a target network is used to evaluate the selected action. The value of.

[0028] By decoupling the action selection and value evaluation steps onto two different networks, DDQN effectively avoids the bootstrapping overestimation problem caused by the same network simultaneously handling selection and evaluation in standard DQN.

[0029] In summary, the data detection method based on hierarchical three-stage deep reinforcement learning of the present invention has the following advantages compared with traditional techniques: 1. Based on the high efficiency of gradient decision trees in tabular data, LightGBM is selected as the base classifier. Compared with computationally intensive deep learning models, LightGBM not only significantly reduces memory usage and computational overhead when processing large-scale structured traffic data, but also achieves accurate extraction and high-performance recognition of traffic features while ensuring training efficiency with its leaf growth strategy. 2. A coarse-to-fine cascaded detection process was designed: the first stage uses a LightGBM binary classifier to quickly filter normal traffic and identify malicious traffic; the second stage uses a LightGBM multi-classifier to initially classify malicious traffic and locate confusing categories with severely overlapping features; the third stage deploys a dedicated classifier for the identified confusing category clusters. This framework effectively alleviates the common problems of class imbalance and long-tail distribution in network traffic by decoupling task complexity layer by layer. 3. A cross-validation-based method for identifying confused categories was designed. In the second stage of training, a 5-fold cross-validation strategy was adopted to obtain the out-of-bag prediction results of the entire sample set. By constructing a global cumulative confusion matrix, the random bias of a single partition was eliminated, ensuring the robustness of misclassification statistics. Based on this, the bidirectional confusion weights between classes were further calculated, and a symmetric class similarity matrix was constructed. Classes were mapped to graph nodes, and bidirectional confusion weights were mapped to edge weights. The graph structure was pruned and filtered by setting a confusion threshold. Finally, connected component analysis was used to identify multiple independent clusters of highly confused categories, and a dedicated fine-grained classifier was constructed for each cluster in the third stage to achieve accurate processing of confused and difficult-to-classify samples. 4. The deep reinforcement learning algorithm DDQN is introduced to adjust the class weights of the LightGBM model for the third stage of confusion class recognition. By introducing deep reinforcement learning, the agent can autonomously discover the efficient weight combination that maximizes the minority class detection performance, thereby overcoming the dependence of traditional optimization methods on manually set search space. At the same time, the experience replay mechanism of DDQN's off-policy strategy is used to maximize the reuse of historical data for learning, avoiding frequent model calls for evaluation and significantly reducing computational overhead.

[0030] The technical method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0031] Figure 1 This presents the overall architecture of a data detection method based on hierarchical three-stage deep reinforcement learning; Figure 2 The specific training process for the LightGBM expert classifier; Figure 3 The algorithm steps for training a confusion-based expert classifier using DDQN; Figure 4 This describes the DDQN network update process. Detailed Implementation

[0032] The technical method of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0033] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0034] Techniques, systems, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the instruction manual.

[0035] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0036] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0037] Example 1 A hierarchical three-stage deep reinforcement learning-based data detection method (H3-DRL) consists of a hierarchical detection architecture with three logical stages. Its data processing flow and decision logic are as follows: Figure 1 As shown.

[0038] An input network traffic sample will be processed according to the following procedure: S1. Obtain traffic data samples Traffic samples Input LightGBM binary classifier Model output vector ; Threshold of LightGBM binary classifier Determine the probability of an attack. Less than Then the traffic data sample This is a benign sample, categorized as ; If the probability of attack Greater than or equal to Then the traffic data sample For suspicious sample set ; S2, Set up suspicious attack samples Input LightGBM multi-classifier The LightGBM multi-classifier obtained a set of suspected attack samples. Predicted category set ; Based on probability vectors Construct a category confusion graph and classify the confused samples based on the category confusion graph. ; S3, Based on the confused sample set Construct a corresponding number of weighted LightGBM expert classifications LightGBM expert classification uses class weights to distinguish the corresponding confused sample subsets, thus obtaining the final confused result set. ; The category weights of the LightGBM expert classification Dynamically optimized by DDQN agents.

[0039] The specific training process for LightGBM expert classification is as follows: Figure 2 As shown.

[0040] The DDQN agent uses the Dual Deep Q-Network (DDQN) algorithm as its core. The detailed algorithm steps are described in [link to article / section]. Figure 3 The text is incomplete and cannot be translated. Figure 4 This is the DDQN network update process.

[0041] Example 2 To comprehensively evaluate the performance and generalization ability of a data detection method based on hierarchical three-stage deep reinforcement learning in different scenarios, this embodiment selects two widely recognized benchmark datasets in the field of intrusion detection: UNSW-NB15 and NSL-KDD, and performs data cleaning, feature type separation and encoding, and random forest-based feature selection preprocessing on the original UNSW-NB15 and NSL-KDD datasets.

[0042] This embodiment comprehensively compares the performance of H3-DRL with a series of mainstream imbalanced learning single-stage models based on LightGBM. The experimental results are shown in Table 1.

[0043] Table 1. Performance comparison of H3-DRL and mainstream imbalance handling methods on the UNSW-NB15 test set.

[0044] The experimental results in Table 1 allow for a deeper analysis of the effectiveness of different imbalance handling strategies. First, the standard LightGBM model achieved the highest accuracy, but its macro-average recall was the worst among all the compared models. This clearly reveals the misleading nature of accuracy in imbalanced scenarios: the model achieves high accuracy by accurately predicting the majority class with a large sample size, but at the cost of sacrificing its ability to detect the minority class, resulting in poor overall balance performance.

[0045] Secondly, this application observes the trade-offs and limitations of traditional SMOTE and cost-sensitive learning methods in improving performance. Compared to standard LGBM, both methods significantly improve macro-average recall, indicating that the model is indeed guided to more actively identify minority class samples. However, this improvement in recall comes at the cost of a huge sacrifice in precision, with both methods achieving a macro-average precision of less than 58%. This shows that, whether through data interpolation or static weight allocation, the model generates a large number of false positives while identifying more minority class samples. This aggressive "better to have too many than too few" strategy ultimately leads to their performance on the overall performance metric Macro-F1-score being even worse than standard LGBM, proving that they are not an ideal solution.

[0046] Ultimately, the H3-DRL framework proposed in this application stood out among all models, demonstrating its superior ability to achieve the best balance between precision and recall. It not only achieved the highest macro-average recall (77.36%), but more importantly, it also maintained the highest macro-average precision (69.64%). This result strongly proves that the "divide and conquer + dynamic optimization" strategy of this application is far superior to simple data sampling or static weighting. By introducing DDQN for intelligent optimization in the third stage, the framework of this application found an optimal balance between "precision" and "recall" for highly confused categories. It is this superior balancing ability that enabled it to achieve a macro-average F1 score of 0.7150, a core metric for measuring overall performance, comprehensively surpassing all single-stage baseline models. This significant advantage in macro-average metrics directly proves that the H3-DRL framework is far more effective than traditional data-level or algorithm-level methods in mitigating performance bias caused by class imbalance.

[0047] Example 3 To demonstrate the effectiveness of the three-stage architecture and the DDQN optimization strategy in handling class imbalance and class confusion, this application designed and conducted a series of ablation experiments. Two variant models were created by removing or replacing key modules of the framework: a single-stage DDQN-LGBM and a three-stage Bayesian-LGBM. All models were evaluated on the same preprocessed dataset, with the core comparison metric being the macro-average F1 score. The results are shown in Table 2, where the attack categories are the original class names of the dataset.

[0048] Table 2 Comparison of macro-average F1 scores of each model on various attack categories in UNSW-NB15 during ablation experiments.

[0049] Note: To verify the model's efficiency in identifying attack traffic, this application excludes the Normal class and adds the standard LGBM as the final reference benchmark in the table.

[0050] Through three sets of comparative experiments, the following conclusions can be drawn from this application: First, by comparing standard LGBM with single-stage DDQN-LGBM, this application finds that adjusting weights using DDQN can improve the classification efficiency of the classifier. Furthermore, the macro-average F1 scores of classes like Backdoor and Worms, which have few scores on the training set, also show significant improvement after DDQN weight adjustment. This demonstrates the feasibility of weight adjustment methods based on deep reinforcement learning.

[0051] Second, by comparing the single-stage DDQN-LGBM with the complete H3-DRL framework of this application, it can be seen that the introduction of the three-stage architecture brings an additional performance gain of 4.32%. Furthermore, the Analysis category, which is completely ignored in the single-stage DDQN-LGBM, can be identified in H3-DRL, demonstrating that structurally separating the detection process can effectively alleviate the problem of class imbalance in identification.

[0052] Third, by comparing the results of the three-stage Bayesian-LGBM and H3-DRL, excluding non-confused categories (since the first two stages are identical, the results for non-confused categories are also identical), H3-DRL significantly improves the efficiency of identifying confused categories. In particular, H3-DRL performs best in the Analysis category, which is difficult for other classifiers to identify, and the Dos category, which is strongly confused with the Exploits category. Ultimately, the improved efficiency in identifying confused categories is reflected in the overall macro-average F1 score, reaching 0.7150. This demonstrates that the second-stage confused category identification method can effectively identify confused categories, thereby alleviating the category confusion problem caused by the inherent similarity of features.

[0053] In summary, this application proposes a joint framework combining structured task decomposition and intelligent policy learning, which significantly improves the performance of intrusion prediction, particularly for minority and confusing classes. The proposed model leverages the closed-loop interaction between the agent and the classification environment. Experimental results provide detailed performance analyses on the UNSW-NB15 and NSL-KDD datasets, as well as comparative analyses of H3-DRL with various baselines. Finally, comparative evaluation results demonstrate that the proposed H3-DRL outperforms other state-of-the-art methods and traditional imbalanced processing techniques such as SMOTE and cost-sensitive learning in terms of prediction performance, especially in terms of macro-average F1 scores.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical methods of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical methods of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical methods to deviate from the spirit and scope of the technical methods of the present invention.

Claims

1. A data detection method based on hierarchical three-stage deep reinforcement learning, characterized in that, Includes the following steps: S1. Obtain traffic data samples Traffic samples Input LightGBM binary classifier Model output vector , This represents a normal probability. Threshold of LightGBM binary classifier Determine the probability of an attack. Less than Then the traffic data sample This is a benign sample, categorized as ; If the probability of attack Greater than or equal to Then the traffic data sample For suspicious sample set ; S2, Transfer suspicious attack samples Input LightGBM multi-classifier The LightGBM multi-classifier obtained a set of suspected attack samples. Predicted category set ; Predicted category set based on all samples Construct a category confusion graph and classify the confused samples based on the category confusion graph. ; S3, Based on the confused sample set Construct a corresponding number of weighted LightGBM expert classifications LightGBM expert classification uses class weights to distinguish the corresponding confused sample subsets, thus obtaining the final confused result set. ; The category weights of the LightGBM expert classification Dynamically optimized by DDQN agents.

2. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 1, characterized in that, Predict the category based on each sample. Construct a category confusion graph and classify the confused samples based on the category confusion graph. The specific content is as follows: The predicted class of each sample is obtained by obfuscating it using an obfuscation recognition mechanism based on 5-fold cross-validation. This leads to the collection of suspicious samples. Confusion matrix on the validation set ; Based on this confusion matrix, the symmetric confusion weights between any two categories can be calculated. ; Based on confusion threshold Identify all category pairs with confusion weights higher than the threshold and construct a category confusion graph; Using category confusion relation graph reasoning to segment suspicious sample sets ; If a suspicious sample is an isolated node in the category confusion graph, it is defined as an attack category that is easily classified. Output the result directly; If a suspicious sample belongs to a connected subgraph consisting of multiple confusing edges in the category confusion graph, then the sample is classified as a confused sample and stored in the confused sample set. middle.

3. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 2, characterized in that, The confusion detection mechanism based on 5-fold cross-validation involves generating an out-of-fold prediction result set, i.e., the predicted class of the sample, for each training sample through 5-fold cross-validation on suspicious attack samples. ; An out-of-fold prediction confusion matrix (OOF) is constructed based on the out-of-fold prediction result set and the true labels of the suspected attack samples. A confusion matrix is ​​constructed based on the true labels of the suspected attack samples and all out-of-place prediction confusion matrices (OOF). .

4. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 3, characterized in that, Symmetric Confusion Weights For category and categories The symmetric obfuscation weights between them are expressed as follows: ； ； ； in, For category Misclassified as a category proportion, For category Misclassified as a category proportion, In the confusion matrix In the above, the number of samples whose true label is category i but were misclassified as category j is: In the confusion matrix In this context, the number of samples with predicted label i but actual label j is denoted as i, where i and j represent two different attack categories.

5. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 4, characterized in that, Based on confusion threshold The specific steps for identifying all category pairs with confusion weights higher than the threshold and constructing a category confusion graph are as follows: The category confusion graph is an undirected weighted graph. ,in, This represents the set of all points in the obfuscated graph, where each point represents an attack category. This represents the set of edges between each point, which is the confusion relationship between each point; Each attack category is a vertex. ; If any two categories and Confusion weights between Higher than In category and categories Add an edge between .

6. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 5, characterized in that, The state space of the DDQN agent is: , including status ; in, ; in, t For the current time step, Indicates the number of categories in the confusion group. Indicate category At the current time step The weight, Indicates the category under the current weight. The macro-average F1 score obtained after training on the current training set. This indicates the current time step. Macro average F1 score for all confusion categories; Through the state space, the DDQN agent obtains the influence of the weight of each category on the macro-average F1 score of each category and the overall macro-average F1 score.

7. The data detection method based on hierarchical three-stage deep reinforcement learning according to claim 6, characterized in that, The action space of the DDQN agent is ; At the current time step The DDQN agent selects an action. Action space Defined as: ； Where m represents the number of actions. Indicates the category to be adjusted. This indicates that the specific operation performed on this class at the current time step is an action. .

8. A data detection method based on hierarchical three-stage deep reinforcement learning according to claim 7, characterized in that, The reward function of the DDQN agent is: ; award The bonus is the increase in the macro average F1 score. The expression is: ； in, It is to perform an action Then, the macro-average F1 score obtained on the validation set at the current time step. This is the macro-average F1 score obtained on the validation set at the previous time step.

9. A data detection method based on hierarchical three-stage deep reinforcement learning according to claim 8, characterized in that, The DDQN agent uses a dual deep Q network as its core algorithm, DDQN. A dual-depth Q-network consists of: a current Q-network used for action selection, whose parameters are... ; The target network used to evaluate the value of actions has the following parameters: .

10. A data detection method based on hierarchical three-stage deep reinforcement learning according to claim 9, characterized in that, During training, the interactive experience of the DDQN agent, namely a series of state transition tuples. It is stored in an experience replay pool. middle; During training, from A small batch of data is randomly sampled from the data. For each sample DDQN minimizes a mean squared error loss function. To update the parameters of the current Q network : ； in, Is the current Q-network pair in state? Next action Value prediction; TD Target The expression is: ； in, It is to perform an action The instant reward obtained afterward It is a discount factor, one between The hyperparameters between these parameters are used to weigh the importance of immediate rewards versus future rewards. The current Q network selects the next state. The most valuable action ,and This section indicates that the target network is used to evaluate the selected action. The value of.

Images