Threat detection method for low-quality labeled data environment

Abstract

The application realizes fine-grained attack classification by effectively utilizing the underlying distribution of existing low-quality label data, discovers the clustering structure in the data, and makes the data either correctly classified under the known label or newly labeled, thereby obtaining high-quality label data. The application fuses the feature and structure information of the original data through the automatic encoder and the graph automatic encoder, estimates the similarity between the sample points and the cluster centers in the potential embedding space through the relative entropy, improves the target distribution precision, and reflects the influence of the number of clusters on the data by combining the data state and the cluster state. The application effectively reduces the problems of model detection accuracy decline, false positive rate increase, and insufficient generalization ability caused by low-quality label data through the data fine classification method, and improves the threat detection precision.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, specifically referring to a threat detection method for low-quality labeled data environments, and particularly involving a graph clustering algorithm based on deep reinforcement learning (DRLGC) to establish an adaptive threat data detection method. Background Technology

[0002] The interconnected nature of networks, while enhancing intelligence in enterprise networks and other environments, also exacerbates cybersecurity risks. Existing network environments require processing massive amounts of data and running complex algorithms, leading to increasingly complex and diverse network interaction environments. This further poses a significant challenge to traditional rule-based or static feature-matching-based network threat detection systems. Current research on network intrusion detection largely focuses on identifying known attack patterns. By collecting and labeling benign and malicious samples, defenders can train classifiers to distinguish attack data. However, existing detection models heavily rely on large-scale, high-quality labeled data, especially deep learning models, whose training performance is closely related to data quality. However, due to factors such as varying levels of expertise among annotators, labeled data often lacks high consistency, affecting model training performance. Furthermore, labeling cybersecurity data is costly, and the continuous evolution of attack behaviors makes comprehensive label coverage a challenge.

[0003] In cybersecurity threat detection, the presence of low-quality labeled data significantly impacts model performance. Existing research has explored various strategies to address this challenge, including supervised learning that relies entirely on labeled data and unsupervised clustering that does not depend on labeled data. However, all these methods have certain limitations and are difficult to fully leverage in complex cybersecurity environments.

[0004] The simplest existing solution is to train the model directly using labeled samples from low-quality labeled data. However, due to the limited availability and inconsistent quality of labeled data, the detection model struggles to accurately learn the decision boundaries of real-world cyberattacks, leading to a decrease in accuracy.

[0005] While existing methods, such as semi-supervised learning, improve a model's ability to learn true categories by combining limited labeled data with a large amount of unlabeled data, these methods require that at least some samples from all categories in the training data be correctly labeled. Meanwhile, zero-shot learning, through knowledge transfer and semantic embedding, enables machine learning models to recognize categories not seen during training. However, these methods rely on fully labeled training datasets; only well-labeled training sets can enable these methods to identify new categories when data is scarce.

[0006] Another approach is to completely ignore label information and directly use unsupervised clustering methods for classification. This method avoids interference from low-quality labels and can, to some extent, discover inherent patterns in the data. However, due to the lack of label guidance, unsupervised methods are generally inferior in classification accuracy to strategies that utilize label information. Furthermore, the excellent performance of existing clustering methods largely depends on an accurate predetermined range of cluster numbers, which is often unavailable in real-world network environments.

[0007] Data annotation plays a crucial role in threat detection. High-quality annotated data can provide richer contextual information, helping threat detection models extract effective features. However, current security data annotation faces the following major problems: First, the incompleteness of annotated data categories. For example, with the continuous emergence of new intrusion attacks, existing annotated data may not cover the latest malicious attack types. Second, the coarse-grained nature of annotation. Due to analysts' limited time or insufficient expertise, some samples may be assigned incorrect or ambiguous labels. For example, in network attack classification tasks, different types of attacks may be incorrectly classified into the same category. Coarse-grained labels not only affect the accuracy of the detection system but also limit the effectiveness of responding to different attacks. Third, the issue of the predetermined number of clusters. Existing anomaly-based systems rely on unsupervised machine learning to detect out-of-distribution samples, but their excellent performance largely depends on an accurately predetermined range of cluster numbers, which is not always available in the real world. In practical applications, obtaining large-scale, high-quality threat data is extremely difficult, especially in the field of cybersecurity, where this challenge is particularly prominent. On the one hand, data annotation is costly; on the other hand, attack techniques are constantly evolving, making annotation work more complex. Furthermore, limited by resources and time, only a small number of data samples can be effectively labeled and utilized, which severely restricts the adaptability and detection capabilities of existing threat detection methods. Summary of the Invention

[0008] This invention aims to detect hidden attacks in the network as accurately as possible through the established detection model, thereby improving the detection performance of the threat detection system and reducing the time required for manual labeling of threat data.

[0009] To achieve the above effects, this invention provides a threat detection method for low-quality labeled data environments. It uses a deep reinforcement learning-based graph clustering (DRLGC) algorithm to establish an adaptive threat data detection method to improve the detection capability of the threat detection system.

[0010] This invention provides a threat detection method for low-quality labeled data environments, comprising: collecting network traffic, representing it as a feature matrix, and using it as training data; constructing a deep reinforcement learning-based graph clustering model as a detection model; training the established detection model using the collected network traffic; establishing an environment in the deep reinforcement learning graph clustering method; determining the number of clusters in the detection model by having a reinforcement learning agent perform different actions; maximizing the accumulated reward function based on the feedback generated by the deep clustering network to form the true number of clusters; labeling the data using existing information; and performing network threat detection. The environment is the object that the agent interacts with, and its state changes under the influence of the agent's actions. Based on the deep clustering model as the environment model in reinforcement learning, different states are generated according to different numbers of clusters given by the agent.

[0011] Preferably, the above method is implemented through a control center, specifically including:

[0012] Step 1: Collect network traffic, represent it as a feature matrix, and use it as training data;

[0013] Step 2: Build a graph clustering method based on deep reinforcement learning;

[0014] Step 3: Use the collected network traffic to train the detection model established in Step 2;

[0015] Step 4: Use existing information to tag the data and perform network threat detection;

[0016] Step 5: Repeat steps 1 to 5 until finished.

[0017] Preferably, step one above includes:

[0018] S101. Measure network traffic and represent it as a traffic feature matrix X. The traffic feature matrix can be represented as: Where d represents the dimension of the traffic data, represents the number of features to be extracted, and n represents the total number of samples;

[0019] S102. Perform standard normalization on the original traffic feature matrix data X, and use it as subsequent training data.

[0020] Preferably, step two above includes:

[0021] S201. Load the dataset X collected in step one and read the traffic feature matrix;

[0022] S202. Based on the key elements of the reinforcement learning decision-making process, the cumulative reward function is maximized according to the feedback generated by the deep clustering network to form the true number of clusters, thus establishing a graph clustering method for deep reinforcement learning.

[0023] S203. Establish the environment in the graph clustering method of deep reinforcement learning. Based on the deep clustering model, use the environment model in reinforcement learning to generate different states according to the different number of clusters given by the agent.

[0024] DRLGC provides the feature matrix X and the original adjacency matrix. The adjacency matrix is ​​structural information built from the neighborhood information of data X; the deep clustering module in DRLGC first integrates information from two sources to obtain a consistent latent representation, and the i-th encoder in AE is represented as follows:

[0025]

[0026] in This represents the output of the i-th encoder layer, where σ represents the non-linear activation function. This represents the learnable parameters of the i-th encoder layer. This represents the offset of the i-th encoder layer. The encoder input in AE is X. The i-th encoder in GAE is represented as follows:

[0027]

[0028] in and Let represent the output of the i-th encoder layer, the learnable parameters, and the bias, respectively. The input to the encoder in GAE is... in It is obtained by further normalizing the original adjacency matrix A. D is the degree matrix of A. in This indicates that each node has a self-joining loop.

[0029] Will and The embedded representations are combined through linear combination operations:

[0030] Z = αZ AE +(1-α)Z GAE (3)

[0031] Where α is a learnable coefficient, automatically adjusted using the gradient descent method, and the importance of the two information sources is selectively determined based on the feedback during training; the DRLGC learns the feature and structural information of the original data by combining AE and GAE.

[0032] By combining the embedding Z as input, the decoders of AE and GAE reconstruct the inputs of the two sub-networks; the i-th decoder in AE is represented as follows:

[0033]

[0034] in and This represents the output, learnable parameters, and bias of the i-th decoder layer;

[0035] The i-th decoder in GAE is represented as follows:

[0036]

[0037] in and Let represent the output, learnable parameters, and bias of the i-th decoder layer, respectively;

[0038] After obtaining the embedding Z, a clustering algorithm is performed on Z to obtain the clustering assignment matrix. in To reinforce the number of data categories determined by the learning agent, K-means(·) is the K-means algorithm; the cluster centers C are further obtained through the clustering assignment matrix and embedding representation, and the cluster centers are calculated by averaging the node embeddings within each cluster;

[0039] S204. Establish the state in the graph clustering method of deep reinforcement learning, and use data embedding to represent Z. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t)}, where Z (t) Let C represent the embedding representation of the t-th deep clustering model iteration. (t) Let represent the cluster center in the t-th iteration of the deep clustering model;

[0040] S205. Establish an agent in a graph clustering method of deep reinforcement learning. The agent selects an action based on the current state and adjusts its strategy based on feedback. Given a state S... (t) The agent outputs the score vector u of the current state. (t) =f Q (S (t) ), f Q (·) represents an agent model for reinforcement learning;

[0041] Z′ (t) =σ(Norm(W) Z Z (t) +b Z (6)

[0042] C′ (t) =σ(Norm(W) C C (t) +b C (7)

[0043] Among them W Z With b Z W C With b C and Norm represent the learnable parameters and biases of the embedded Z subnetwork, the learnable parameters and biases of the cluster center C subnetwork, and the normalized network layer, respectively; Z′ (t) and C′ (t) These represent the outputs of the corresponding sub-networks; the agent network concatenates the two learned parameters to calculate the score for the current state, as shown below:

[0044] u (t) =σ S (W S (Concat([Z′ (t) ,C′ (t) ]))+b S (8)

[0045] Where σ S W S b S Concat and represent the softmax function, which outputs the learnable parameters and biases of the network, and the concatenation function, respectively. Through the combination of the above, the agent fuses the state of the original data with the state of the cluster center, and performs iterative learning according to different actions to achieve automated cluster number determination.

[0046] S206. Establish the action strategy in the graph clustering method of deep reinforcement learning. DRLGC establishes a greedy strategy for reinforcement learning based on formula (9). The greedy strategy shown in formula (9) evaluates different cluster numbers and makes intelligent decisions during training, where the greedy parameter ε increases with the number of iterations.

[0047]

[0048] The `random()` function represents randomly selecting an action from the action space.

[0049] S207. Establish a state transition strategy in graph clustering methods for deep reinforcement learning. After the agent makes an action based on the current state, the environment's current state S... (t) It will transform into S at the next moment. (t+1) ;

[0050] S208. Establish the reward function in the graph clustering method of deep reinforcement learning by minimizing the distance between each node and its corresponding cluster center. To improve inter-class similarity, the distance between different cluster centers is maximized. To improve category discrimination ability, the reward function is expressed as follows:

[0051]

[0052] By maximizing the reward function, the agent can make the most accurate judgment.

[0053] Preferably, the key elements of the above-mentioned reinforcement learning decision-making process include environment, state, agent, action, state transition, and reward function.

[0054] Preferably, step three above includes:

[0055] S301. Establish a DRLGC model training method that includes training of a deep clustering network and training of an agent. The deep clustering network is trained in a self-supervised manner, and the agent adopts an experience replay training strategy, establishing an experience buffer by using a four-tuple of current state, current action, current reward, and next state. Where t s This is the buffer's initial iteration state, l t The buffer limit width is [value], and the number of iterations in the deep clustering network is [value]. Based on the collected empirical buffer, the loss L will be minimized. Q To train the agent and make the agent f Q Take the optimal action; the loss function is expressed as follows:

[0056]

[0057] Where η is the discount factor; where the first two terms in formula (13) represent the calculation of the target value, including the discount estimate of the immediate reward and the future optimal value, and the last term represents the value prediction of the current action, which is used to compare with the target value. The overall goal of the loss function is to make the predicted value close to the real action value by minimizing the square of the temporal difference error.

[0058] S302. Train the DRLGC model to obtain the final number of data clusters. Deep clustering models are used to simulate changes in the current environment, state S (t) It uses a deep clustering model based on the current action Generate; Reward R (t) The result is calculated based on each node and its corresponding cluster center, and then further based on the reward R. (t) The reinforcement learning agent is trained with the experience pool to establish the DRLGC model;

[0059] The depth map clustering method integrates data features and their neighbor information to obtain the embedding representation of the t-th iteration using a spatial feature fusion approach. Where d′ represents the number of features after dimensionality reduction, and f EThe encoder in the depth map clustering algorithm is a combination of the encoders from AE and GAE; in the action Clustering results obtained under the guidance of deep clustering loss. Furthermore, the obtained clusters are mapped to a low-dimensional space, preserving the similarity of data points within clusters and the differences between clusters, thus obtaining the cluster center representation.

[0060] Using data embedding to represent Z (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) The status will include node information and cluster information of the original data;

[0061] Based on the established state information, DRLGC further establishes a reinforcement learning agent f. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. The greedy strategy shown in formula (9) is used to evaluate different cluster numbers and make intelligent decisions during training; the final number of clusters is obtained through multiple iterations.

[0062] Preferably, step four above includes:

[0063] S401. The original data undergoes fine-grained classification and labeling. For existing low-quality labeled data and the obtained clustering results, the fine-grained labeling method based on clustering-label co-mapping follows these guidelines:

[0064] (1) When the clustering results It contains only data of a single known category Y j When (j=1,…,m)∈Y, a deterministic mapping relation is established, i.e., P i Belongs to category Y j Y represents the low-quality labeled data of the original data, which contains only partial labels and may not be accurate. This represents the final calculated number of clusters;

[0065] (2) When there is a missing category, there exists Then determine P i For potential new classes;

[0066] (3) In a coarse-grained scenario, there exists P i With P jWhen all clusters contain the same single-category label, it is determined that a coarse-grained label exists, and a new label is created for the cluster set that contains the fewest instances of that single-category label.

[0067] (4) In mixed category scenarios, i.e., clustering P i When there are two or more labels, this embodiment learns from the existing label data through a decision tree and performs clustering P. i The samples in the data are relabeled; the clustering-label co-mapping operation is repeated multiple times to make the number of data types obtained by label matching as equal as possible to the number of clusters calculated by DRLGC;

[0068] S402. Use the encoder f that has already been trained in the DRLGC model. E The detection model is trained using the Decision Forest method to complete threat detection.

[0069] Preferably, the above method integrates data features and their neighbor information to obtain the embedding representation Z of the t-th iteration using a spatial feature fusion approach. (t) In action Clustering results are obtained under the guidance of deep clustering loss, and the obtained clusters are mapped to a low-dimensional space; the similarity of data points within clusters and the differences between clusters are preserved to obtain the cluster center representation C. (t) Z is represented using data embedding. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) The state will include node information and cluster information of the original data; based on the established state information, a reinforcement learning agent f will be further established based on a graph clustering model of deep reinforcement learning. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. Finally, a greedy strategy evaluates different clustering numbers and makes intelligent decisions during training; the final number of clusters is obtained through multiple iterations.

[0070] The present invention also provides a system for implementing the above-described threat detection method for low-quality labeled data environments, comprising:

[0071] The data acquisition module is used to collect network traffic, represent it as a feature matrix, and use it as training data.

[0072] The graph clustering building module is used to build graph clustering methods based on deep reinforcement learning;

[0073] The detection model training module is used to train and establish a detection model using the collected network traffic.

[0074] The network threat detection module is used to tag data and detect network threats using existing information.

[0075] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0076] This invention presents a threat detection method for low-quality labeled data environments. It achieves fine-grained attack classification by effectively utilizing the underlying distribution of existing low-quality labeled data. By discovering cluster structures in the data, it ensures that the data is either correctly classified under known labels or generates entirely new labels, thus obtaining high-quality labeled data. First, it fuses the feature and structural information of the original data using an autoencoder (AE) and a graph autoencoder (GAE). It also improves the accuracy of the target distribution by estimating the similarity between sample points and cluster centers in the latent embedding space using relative entropy. Then, it combines data state and cluster state to reflect the impact of the number of clusters on the data. Finally, this embodiment unifies cluster number determination and unsupervised representation learning into a single framework using reinforcement learning, and determines the final data structure through greedy actions. By correctly recovering the cluster structure in the original low-quality dataset, DRLGC generates high-quality labeled data for security fields such as threat detection. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This diagram illustrates the framework of the threat detection method for low-quality labeled data environments according to the present invention.

[0079] Figure 2 A schematic diagram of graph clustering based on deep reinforcement learning in this invention is shown;

[0080] Figure 3 The figure shows an embodiment of the threat detection method of the present invention for low-quality labeled data environments. Detailed Implementation

[0081] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the invention.

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0083] like Figure 1 As shown, this invention provides an embodiment of a threat detection method for low-quality labeled data environments, comprising:

[0084] The first step is to collect raw network traffic data and real-time network traffic data from embedded devices;

[0085] The second step is to model a graph clustering model based on deep reinforcement learning.

[0086] The third step is to train a graph clustering model based on deep reinforcement learning.

[0087] The fourth step is to relabel and align the low-quality data and train the threat detection model.

[0088] like Figure 2 As shown, the overall iterative process of the graph clustering model based on deep reinforcement learning is as follows:

[0089] Specifically, the depth map clustering method in this embodiment integrates data features and their neighbor information to obtain the embedding representation Z of the t-th iteration using a spatial feature fusion approach. (t) This embodiment describes the action. Clustering results are obtained under the guidance of deep clustering loss, and the obtained clusters are further mapped to a low-dimensional space while trying to maintain the similarity of data points within clusters and the differences between clusters, thus obtaining the cluster center representation C. (t) To capture both local and global information from the raw data, this embodiment utilizes the data embedding representation Z. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) In this way, the state will contain node information and cluster information of the original data. Based on the established state information, a reinforcement learning agent f is further established based on a graph clustering model of deep reinforcement learning. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. Finally, a greedy strategy evaluates different clustering numbers and makes intelligent decisions during training. The final number of clusters is obtained through multiple iterations.

[0090] The present invention also provides an embodiment, including the following steps:

[0091] Step 1: In the control center, collect network traffic, represent it as a feature matrix, and use it as training data.

[0092] Step 2: In the control center, build a graph clustering method based on deep reinforcement learning.

[0093] Step 3: In the control center, the detection model established in Step 2 is trained using the collected network traffic.

[0094] Step 4: In the control center, use the existing information to tag the data and perform network threat detection.

[0095] Step 5: End.

[0096] The feature matrix setting described in step one specifically includes the following steps:

[0097] Step A: In the control center, measure the network traffic and represent it as a traffic feature matrix X.

[0098] End-to-end network traffic, called origin-to-destination (OD) flow, describes the traffic from the origin node to the destination node. For all possible OD pairs in a network with N nodes, a traffic matrix can be used. This traffic matrix is ​​then extracted using a network security dataset traffic feature extraction tool (Cicflowmeter) to form a traffic feature matrix. The traffic feature matrix can be represented as follows: Where d represents the dimension of the traffic data (indicating the number of features to be extracted), and n represents the total number of samples.

[0099] Step B: At the control center, the original traffic feature matrix data X is normalized and used as subsequent training data.

[0100] The graph clustering method based on deep reinforcement learning described in step two specifically includes the following steps:

[0101] Step A: In the control center, load the dataset X collected in Step 1 and read the traffic feature matrix.

[0102] Step B: In the control center, establish a deep reinforcement learning graph clustering method.

[0103] Determining the cluster number requires the reinforcement learning agent to maximize the cumulative reward function by implementing different actions and based on the feedback generated by the deep clustering network, thereby forming the true cluster number. Therefore, the key elements forming the reinforcement learning decision-making process include environment, state, agent, action, state transition, and reward function.

[0104] Step C: In the control center, set up the environment for the graph clustering method of deep reinforcement learning.

[0105] The environment is the object that the agent interacts with, and its state changes in response to the agent's actions. DRLGC uses a deep clustering model as the environment model in reinforcement learning, generating different states based on the different numbers of clusters provided by the agent.

[0106] Given the feature matrix X and the original adjacency matrix The adjacency matrix is ​​structural information built from the neighborhood information of data X. The deep clustering module in DRLGC first integrates information from two sources to obtain a consistent latent representation. The i-th encoder in AE is represented as follows:

[0107]

[0108] in This represents the output of the i-th encoder layer, where σ represents the non-linear activation function. This represents the learnable parameters of the i-th encoder layer. Let X represent the offset of the i-th encoder layer. The encoder input in the AE is X. The i-th encoder in the GAE is represented as follows:

[0109]

[0110] in and These represent the output of the i-th encoder layer, the learnable parameters, and the bias, respectively. The input to the encoder in GAE is... in It is obtained by further normalizing the original adjacency matrix A. D is the degree matrix of A. in This indicates that each node has a self-joining loop.

[0111] Then, and The embedded representations are combined through linear combination operations:

[0112] Z = αZ AE +(1-α)Z GAE (3)

[0113] Here, α is a learnable coefficient that can be automatically adjusted using gradient descent. It selectively determines the importance of the two information sources based on feedback during training. DRLGC improves clustering performance by combining AE and GAE to learn the feature and structural information of the original data.

[0114] Furthermore, by combining the embedding Z as input, the decoders of AE and GAE reconstruct the inputs of the two subnetworks. Unlike existing methods, the inputs of the two subnetworks are reconstructed using a consistent latent representation. The i-th decoder in AE is represented as follows:

[0115]

[0116] in and This represents the output of the i-th decoder layer, which contains learnable parameters and biases. The i-th decoder in the GAE is represented as follows:

[0117]

[0118] in and represents the output of the i-th decoder layer, the learnable parameters, and the bias, respectively.

[0119] After obtaining the embedding Z, a clustering algorithm is performed on Z to obtain the clustering assignment matrix. in The number of data categories determined by the reinforcement learning agent is determined by step E, where K-means(·) is the K-means algorithm. Cluster centers C can be further obtained through the cluster assignment matrix and embedding representation; these centers are calculated by averaging the node embeddings within each cluster. Since clustering is an unsupervised task, the deep clustering network is trained in a unified optimization manner using relative entropy combined with contrastive loss and mean squared error.

[0120] Step D: In the control center, establish the state in the graph clustering method of deep reinforcement learning.

[0121] A state is a description of the environment at a given moment, containing information that the agent can currently perceive. This embodiment utilizes data embedding to represent Z. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t)}. Z (t) Let C represent the embedding representation of the t-th deep clustering model iteration. (t) This represents the cluster center in the t-th iteration of the deep clustering model.

[0122] Step E: In the control center, establish the agent in the graph clustering method of deep reinforcement learning.

[0123] The agent selects an action based on the current state and adjusts its strategy based on feedback. Given a state S (t) The agent outputs the score vector u of the current state. (t) =f Q (S (t) ), f Q (·) represents the agent model of reinforcement learning. Specifically, the state is embedded into the following two sub-networks.

[0124] Z′ (t) =σ(Norm(W) Z Z (t) +b Z (6)

[0125] C′ (t) =σ(Norm(W) C C (t) +b C (7)

[0126] Among them W Z With b Z W C With b C and Norm represent the learnable parameters and biases of the embedded Z subnetwork, the learnable parameters and biases of the cluster center C subnetwork, and the normalized network layer, respectively. Z′ (t) and C′ (t) These represent the outputs of the corresponding sub-networks. Then, the agent network concatenates the two learned parameters to calculate the score for the current state, as shown below:

[0127] u (t) =σ S (W S (Concat([Z′(t) ,C′ (t) ]))+b S (8)

[0128] Where σ S W S b S `and Concat` represent the softmax function, the learnable parameters and biases of the output network, and the concatenation function, respectively. Through this combination, the agent can fuse the state of the original data with the state of the cluster centers and iteratively learn based on different actions, achieving automated cluster number determination.

[0129] Step F: In the control center, establish the action policy in the graph clustering method of deep reinforcement learning.

[0130] An action is an operation chosen by the agent based on the current state, and the action chosen by the agent will affect the state of the environment. DRLGC establishes a greedy policy for reinforcement learning based on formula (9). The greedy policy shown in formula (9) evaluates different numbers of clusters and makes intelligent decisions during training, where the greedy parameter ε increases with the number of iterations.

[0131]

[0132] The random() function represents randomly selecting an action from the action space.

[0133] Step G: In the control center, establish the state transition strategy in the graph clustering method of deep reinforcement learning.

[0134] After the agent performs an action based on the current state, the current state S of the environment... (t) It will transform into S at the next moment. (t+1) During the learning process, changes in actions and the optimization process of the deep clustering model in DRLGC by minimizing the loss will cause state transitions.

[0135] Step H: In the control center, establish the reward function in the graph clustering method of deep reinforcement learning.

[0136] A reward is the feedback an agent receives based on the changes its chosen action causes to the environment. In state S... (t) In this situation, the intelligent agent gives an action. In this process, the environment will generate a corresponding reward function and feed it back to the agent network. The generated reward function needs to reflect whether the current action can increase cohesion within the same cluster and separate different clusters. Therefore, by minimizing the distance between each node and its corresponding cluster center... To improve inter-class similarity, the distance between different cluster centers is maximized. To improve category discrimination ability, the reward function is expressed as follows:

[0137]

[0138] By maximizing the reward function, the agent can make the most accurate judgment.

[0139] The training of the detection model described in step three specifically includes the following steps:

[0140] Step A: In the control center, establish the model training method for DRLGC.

[0141] The DRLGC model proposed in this embodiment mainly consists of two parts: training the deep clustering network and training the agent. The deep clustering network is trained in a self-supervised manner. For the agent network, this embodiment adopts an experience replay training strategy. Specifically, during the training process, this embodiment establishes an experience buffer by creating a four-tuple of current state, current action, current reward, and next state. Where t s This is the buffer's initial iteration state, l t is the buffer limit width, and is the number of iterations in the deep clustering network. Based on the collected empirical buffer, the loss L will be minimized. Q To train the agent and make the agent f Q Take the optimal action. The loss function is expressed as follows:

[0142]

[0143] Where η is the discount factor. The first two terms in formula (13) represent the calculation of the target value, including the discount estimate of the immediate reward and the future optimal value. The last term represents the value prediction of the current action, used for comparison with the target value. The overall goal of the loss function is to make the predicted value approximate the true action value by minimizing the square of the temporal difference error.

[0144] Step B: In the control center, train the DRLGC model to obtain the final number of data clusters.

[0145] Deep clustering models are used to simulate changes in the current environment, state S (t) It uses a deep clustering model based on the current action Generate. Reward R (t) The result is calculated based on each node and its corresponding cluster center, and then further based on the reward R. (t) The reinforcement learning agent is trained with the experience pool to establish the DRLGC model.

[0146] Specifically, the depth map clustering method in this embodiment integrates data features and their neighbor information to obtain the embedding representation of the t-th iteration using a spatial feature fusion approach. Where d′ represents the number of features after dimensionality reduction, and f E This represents the encoder in the depth map clustering algorithm, which is a combination of the encoders from AE and GAE. This embodiment describes the action... Clustering results obtained under the guidance of deep clustering loss. Furthermore, the obtained clusters are mapped to a low-dimensional space while preserving as much similarity as possible between data points within clusters and differences between clusters, thus obtaining the cluster center representation. To capture both local and global information from the original data, this embodiment utilizes the data embedding representation Z. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) In this way, the state will contain node information and cluster information of the original data. Based on the established state information, DRLGC further establishes a reinforcement learning agent f. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. Finally, the greedy strategy shown in formula (9) is used to evaluate different cluster numbers and make intelligent decisions during training. The final number of clusters is obtained through multiple iterations.

[0147] Step four, data labeling and threat detection, specifically includes the following steps:

[0148] Step A: In the control center, the raw data is further categorized.

[0149] In response to the existing low-quality labeled data and the obtained clustering results, this embodiment further implements label matching operations to achieve finer-grained data classification. The fine-grained labeling method based on clustering-label co-mapping proposed in this embodiment follows the following criteria: (1) When the clustering results It contains only data of a single known category Y j When (j=1,…,m)∈Y, a deterministic mapping relation is established, i.e., P i Belongs to category Y j Y represents the low-quality labeled data of the original data, which contains only partial labels and may not be accurate. (2) In the case of missing categories, there exists Then determine P i(3) In coarse-grained scenarios, there exists P i With P j When all clusters have the same single-category label, it is determined that there is a coarse-grained label, and a new label is created for the cluster set that contains the fewest single-category labels; (4) In the case of mixed categories, i.e. cluster P i When there are two or more labels, this embodiment learns from the existing label data through a decision tree and performs clustering P. i The samples are relabeled. The clustering-label co-mapping operation is repeated multiple times to ensure that the number of data types obtained from label matching is as equal as possible to the number of clusters calculated by DRLGC. Real-world low-quality labeled data is a complex situation involving a mixture of various scenarios. The label matching operation implemented in this embodiment aims to classify the data as finely as possible, minimizing the workload of manual verification while maintaining semantic consistency with expert annotations.

[0150] Step B: In the control center, the raw data is further categorized.

[0151] Regarding threat detection, this embodiment directly uses the encoder f that has already been trained in the DRLGC model. E We combine the Decision Forest method to train a detection model similar to traditional methods.

[0152] Step 5: End.

[0153] This invention also provides an embodiment of a threat detection method for low-quality labeled data environments, comprising: collecting network traffic, representing it as a feature matrix, and using it as training data; constructing a deep reinforcement learning-based graph clustering model as a detection model; training the established detection model using the collected network traffic; establishing an environment in the deep reinforcement learning graph clustering method; determining the number of clusters in the detection model by having a reinforcement learning agent perform different actions; maximizing the accumulated reward function based on the feedback generated by the deep clustering network to form the true number of clusters; labeling the data using existing information and performing network threat detection. The environment is the object that the agent interacts with, and its state changes under the influence of the agent's actions. Based on the deep clustering model as the environment model in reinforcement learning, different states are generated according to different numbers of clusters given by the agent.

[0154] like Figure 3 As shown, this invention also provides an embodiment of a threat detection method for low-quality labeled data environments, implemented through a control center, the method specifically including:

[0155] Step 1: Collect network traffic, represent it as a feature matrix, and use it as training data;

[0156] Step 2: Build a graph clustering method based on deep reinforcement learning;

[0157] Step 3: Use the collected network traffic to train the detection model established in Step 2;

[0158] Step 4: Use existing information to tag the data and perform network threat detection;

[0159] Step 5: Repeat steps 1 to 5 until finished.

[0160] In some embodiments, step one includes:

[0161] S101. Measure network traffic and represent it as a traffic feature matrix X. The traffic feature matrix can be represented as: Where d represents the dimension of the traffic data, represents the number of features to be extracted, and n represents the total number of samples;

[0162] S102. Perform standard normalization on the original traffic feature matrix data X, and use it as subsequent training data.

[0163] In some embodiments, step two includes:

[0164] S201. Load the dataset X collected in step one and read the traffic feature matrix;

[0165] S202. Based on the key elements of the reinforcement learning decision-making process, the cumulative reward function is maximized according to the feedback generated by the deep clustering network to form the true number of clusters, thus establishing a graph clustering method for deep reinforcement learning.

[0166] S203. Establish the environment in the graph clustering method of deep reinforcement learning. Based on the deep clustering model, use the environment model in reinforcement learning to generate different states according to the different number of clusters given by the agent.

[0167] DRLGC provides the feature matrix X and the original adjacency matrix. The adjacency matrix is ​​structural information built from the neighborhood information of data X; the deep clustering module in DRLGC first integrates information from two sources to obtain a consistent latent representation, and the i-th encoder in AE is represented as follows:

[0168]

[0169] in This represents the output of the i-th encoder layer, where σ represents the non-linear activation function. This represents the learnable parameters of the i-th encoder layer. This represents the offset of the i-th encoder layer. The encoder input in AE is X. The i-th encoder in GAE is represented as follows:

[0170]

[0171] in and Let represent the output of the i-th encoder layer, the learnable parameters, and the bias, respectively. The input to the encoder in GAE is... in It is obtained by further normalizing the original adjacency matrix A. D is the degree matrix of A. in This indicates that each node has a self-joining loop.

[0172] Will and The embedded representations are combined through linear combination operations:

[0173] Z = αZ AE +(1-α)Z GAE (3)

[0174] Where α is a learnable coefficient, automatically adjusted using the gradient descent method, and the importance of the two information sources is selectively determined based on the feedback during training; the DRLGC learns the feature and structural information of the original data by combining AE and GAE.

[0175] By combining the embedding Z as input, the decoders of AE and GAE reconstruct the inputs of the two sub-networks; the i-th decoder in AE is represented as follows:

[0176]

[0177] in and This represents the output, learnable parameters, and bias of the i-th decoder layer;

[0178] The i-th decoder in GAE is represented as follows:

[0179]

[0180] in and Let represent the output, learnable parameters, and bias of the i-th decoder layer, respectively;

[0181] After obtaining the embedding Z, a clustering algorithm is performed on Z to obtain the clustering assignment matrix. in To reinforce the number of data categories determined by the learning agent, K-means(·) is the K-means algorithm; the cluster centers C are further obtained through the clustering assignment matrix and embedding representation, and the cluster centers are calculated by averaging the node embeddings within each cluster;

[0182] S204. Establish the state in the graph clustering method of deep reinforcement learning, and use data embedding to represent Z. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t)}, where Z (t) Let C represent the embedding representation of the t-th deep clustering model iteration. (t) Let represent the cluster center in the t-th iteration of the deep clustering model;

[0183] S205. Establish an agent in a graph clustering method of deep reinforcement learning. The agent selects an action based on the current state and adjusts its strategy based on feedback. Given a state S... (t) The agent outputs the score vector u of the current state. (t) =f Q (S (t) ), f Q (·) represents an agent model for reinforcement learning;

[0184] Z′ (t) =σ(Norm(W) Z Z (t) +b Z (6)

[0185] C′ (t) =σ(Norm(W) C C (t) +b C (7)

[0186] Among them W Z With b Z W C With b C and Norm represent the learnable parameters and biases of the embedded Z subnetwork, the learnable parameters and biases of the cluster center C subnetwork, and the normalized network layer, respectively; Z′ (t) and C′ (t) These represent the outputs of the corresponding sub-networks; the agent network concatenates the two learned parameters to calculate the score for the current state, as shown below:

[0187] u (t) =σ S (W S (Concat([Z′ (t) ,C′ (t) ]))+b S (8)

[0188] Where σ S W S b SConcat and represent the softmax function, which outputs the learnable parameters and biases of the network, and the concatenation function, respectively. Through the combination of the above, the agent fuses the state of the original data with the state of the cluster center, and performs iterative learning according to different actions to achieve automated cluster number determination.

[0189] S206. Establish the action strategy in the graph clustering method of deep reinforcement learning. DRLGC establishes a greedy strategy for reinforcement learning based on formula (9). The greedy strategy shown in formula (9) evaluates different cluster numbers and makes intelligent decisions during training, where the greedy parameter ε increases with the number of iterations.

[0190]

[0191] The `random()` function represents randomly selecting an action from the action space.

[0192] S207. Establish a state transition strategy in graph clustering methods for deep reinforcement learning. After the agent makes an action based on the current state, the environment's current state S... (t) It will transform into S at the next moment. (t+1) ;

[0193] S208. Establish the reward function in the graph clustering method of deep reinforcement learning by minimizing the distance between each node and its corresponding cluster center. To improve inter-class similarity, the distance between different cluster centers is maximized. To improve category discrimination ability, the reward function is expressed as follows:

[0194]

[0195] By maximizing the reward function, the agent can make the most accurate judgment.

[0196] In some embodiments, key elements of the reinforcement learning decision-making process include environment, state, agent, action, state transition, and reward function.

[0197] In some embodiments, step three includes:

[0198] S301. Establish a DRLGC model training method that includes training of a deep clustering network and training of an agent. The deep clustering network is trained in a self-supervised manner, and the agent adopts an experience replay training strategy, establishing an experience buffer by using a four-tuple of current state, current action, current reward, and next state. Where t s This is the buffer's initial iteration state, l tThe buffer limit width is [value], and the number of iterations in the deep clustering network is [value]. Based on the collected empirical buffer, the loss L will be minimized. Q To train the agent and make the agent f Q Take the optimal action; the loss function is expressed as follows:

[0199]

[0200] Where η is the discount factor; where the first two terms in formula (13) represent the calculation of the target value, including the discount estimate of the immediate reward and the future optimal value, and the last term represents the value prediction of the current action, which is used to compare with the target value. The overall goal of the loss function is to make the predicted value close to the real action value by minimizing the square of the temporal difference error.

[0201] S302. Train the DRLGC model to obtain the final number of data clusters. Deep clustering models are used to simulate changes in the current environment, state S (t) It uses a deep clustering model based on the current action Generate; Reward R (t) The result is calculated based on each node and its corresponding cluster center, and then further based on the reward R. (t) The reinforcement learning agent is trained with the experience pool to establish the DRLGC model;

[0202] The depth map clustering method integrates data features and their neighbor information to obtain the embedding representation of the t-th iteration using a spatial feature fusion approach. Where d′ represents the number of features after dimensionality reduction, and f E The encoder in the depth map clustering algorithm is a combination of the encoders from AE and GAE; in the action Clustering results obtained under the guidance of deep clustering loss. Furthermore, the obtained clusters are mapped to a low-dimensional space, preserving the similarity of data points within clusters and the differences between clusters, thus obtaining the cluster center representation.

[0203] Using data embedding to represent Z (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) The status will include node information and cluster information of the original data;

[0204] Based on the established state information, DRLGC further establishes a reinforcement learning agent f. Q (S (t) ), in the reward function R (t) With agent loss LQ Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. The greedy strategy shown in formula (9) is used to evaluate different cluster numbers and make intelligent decisions during training; the final number of clusters is obtained through multiple iterations.

[0205] In some embodiments, step four includes:

[0206] S401. The original data undergoes fine-grained classification and labeling. For existing low-quality labeled data and the obtained clustering results, the fine-grained labeling method based on clustering-label co-mapping follows these guidelines:

[0207] (1) When the clustering results It contains only data of a single known category Y j When (j=1,…,m)∈Y, a deterministic mapping relation is established, i.e., P i Belongs to category Y j Y represents the low-quality labeled data of the original data, which contains only partial labels and may not be accurate. This represents the final calculated number of clusters;

[0208] (2) When there is a missing category, there exists Then determine P i For potential new classes;

[0209] (3) In a coarse-grained scenario, there exists P i With P j When all clusters contain the same single-category label, it is determined that a coarse-grained label exists, and a new label is created for the cluster set that contains the fewest instances of that single-category label.

[0210] (4) In mixed category scenarios, i.e., clustering P i When there are two or more labels, this embodiment learns from the existing label data through a decision tree and performs clustering P. i The samples in the data are relabeled; the clustering-label co-mapping operation is repeated multiple times to make the number of data types obtained by label matching as equal as possible to the number of clusters calculated by DRLGC;

[0211] S402. Use the encoder f that has already been trained in the DRLGC model. E The detection model is trained using the Decision Forest method to complete threat detection.

[0212] In some embodiments, the method integrates data features and their neighbor information to obtain the embedding representation Z for the t-th iteration using a spatial feature fusion approach. (t) In action Clustering results are obtained under the guidance of deep clustering loss, and the obtained clusters are mapped to a low-dimensional space; the similarity of data points within clusters and the differences between clusters are preserved to obtain the cluster center representation C. (t) Z is represented using data embedding. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) The state will include node information and cluster information of the original data; based on the established state information, a reinforcement learning agent f will be further established based on a graph clustering model of deep reinforcement learning. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. Finally, a greedy strategy evaluates different clustering numbers and makes intelligent decisions during training; the final number of clusters is obtained through multiple iterations.

[0213] The present invention also provides a system embodiment for implementing the above-described threat detection method for low-quality labeled data environments, comprising:

[0214] The data acquisition module is used to collect network traffic, represent it as a feature matrix, and use it as training data.

[0215] The graph clustering building module is used to build graph clustering methods based on deep reinforcement learning;

[0216] The detection model training module is used to train and establish a detection model using the collected network traffic.

[0217] The network threat detection module is used to tag data and detect network threats using existing information.

[0218] The present invention also provides an embodiment of a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0219] Compared with the prior art, the present invention has the following advantages:

[0220] (1) Existing detection methods rely on predefined cluster numbers and complete label data. This invention unifies data structure determination and unsupervised learning methods into a framework through reinforcement learning mechanism, and uses the DRLGC method to address the problem of low-quality data labels in threat detection, which can effectively discover hidden threats.

[0221] (2) It can provide early warnings on the system status, thereby better preventing potential risks and security issues.

[0222] (3) It has high detection accuracy and the ability to label complex data.

[0223] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0224] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0225] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0227] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0228] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0229] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0230] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0231] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0232] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0233] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0234] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A threat detection method for low-quality labeled data environments, characterized in that... The method includes: collecting network traffic, representing it as a feature matrix, and using it as training data; constructing a deep reinforcement learning-based graph clustering model as a detection model; training the established detection model using the collected network traffic; establishing an environment within the deep reinforcement learning graph clustering method; determining the number of clusters in the detection model by having a reinforcement learning agent perform different actions; maximizing the accumulated reward function based on feedback generated by the deep clustering network to form the true number of clusters; labeling the data using existing information; and performing network threat detection. The environment is the object through which the agent interacts, changing its state under the influence of the agent's actions. Based on the deep clustering model as the environment model in reinforcement learning, different states are generated according to different numbers of clusters provided by the agent.

2. The threat detection method for low-quality labeled data environments according to claim 1, implemented through a control center, is characterized in that... The method specifically includes: Step 1: Collect network traffic, represent it as a feature matrix, and use it as training data; Step 2: Build a graph clustering method based on deep reinforcement learning; Step 3: Use the collected network traffic to train the detection model established in Step 2; Step 4: Use existing information to tag the data and perform network threat detection; Step 5: Repeat steps 1 to 5 until finished.

3. The threat detection method for low-quality labeled data environments according to claim 2, characterized in that... Step one includes: S101. Measure network traffic and represent it as a traffic feature matrix X. The traffic feature matrix can be represented as: Where d represents the dimension of the traffic data, represents the number of features to be extracted, and n represents the total number of samples; S102. Perform standard normalization on the original traffic feature matrix data X, and use it as subsequent training data.

4. The threat detection method for low-quality labeled data environments according to claim 2, characterized in that, Step two includes: S201. Load the dataset X collected in step one and read the traffic feature matrix; S202. Based on the key elements of the reinforcement learning decision-making process, a graph clustering method for deep reinforcement learning is established by maximizing the accumulated reward function according to the feedback generated by the deep clustering network to form the true number of clusters. S203. Establish the environment in the graph clustering method of deep reinforcement learning. Based on the deep clustering model, use the environment model in reinforcement learning to generate different states according to the different number of clusters given by the agent. DRLGC provides the feature matrix X and the original adjacency matrix. The adjacency matrix is ​​structural information built from the neighborhood information of data X; the deep clustering module in DRLGC first integrates information from two sources to obtain a consistent latent representation, and the i-th encoder in AE is represented as follows: in This represents the output of the i-th encoder layer, where σ represents the non-linear activation function. This represents the learnable parameters of the i-th encoder layer. This represents the offset of the i-th encoder layer. The encoder input in AE is X. The i-th encoder in GAE is represented as follows: in and Let represent the output of the i-th encoder layer, the learnable parameters, and the bias, respectively. The input to the encoder in GAE is... in It is obtained by further normalizing the original adjacency matrix A. D is the degree matrix of A. in This indicates that each node has a self-joining loop. AE and IGAE The embedded representations are combined through linear combination operations: Z=αZ AE +(1-α)Z GAE (3) Where α is a learnable coefficient, automatically adjusted using the gradient descent method, and the importance of the two information sources is selectively determined based on the feedback during training; the DRLGC learns the feature and structural information of the original data by combining AE and GAE. By combining the embedding Z as input, the decoders of AE and GAE reconstruct the inputs of the two sub-networks; the i-th decoder in AE is represented as follows: in and This represents the output, learnable parameters, and bias of the i-th decoder layer; The i-th decoder in GAE is represented as follows: in and Let represent the output, learnable parameters, and bias of the i-th decoder layer, respectively; After obtaining the embedding Z, a clustering algorithm is performed on Z to obtain the clustering assignment matrix. in To determine the number of data categories for the reinforcement learning agent, K-means(·) is the K-means algorithm; the cluster centers C are further obtained through the clustering assignment matrix and embedding representation, and the cluster centers are calculated by averaging the node embeddings within each cluster; S204. Establish the state in the graph clustering method of deep reinforcement learning, and use data embedding to represent Z. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) }, where Z (t) Let C represent the embedding representation of the t-th deep clustering model iteration. (t) Let represent the cluster center in the t-th iteration of the deep clustering model; S205. Establish an agent in a graph clustering method of deep reinforcement learning. The agent selects an action based on the current state and adjusts its strategy based on feedback. Given a state S... (t) The agent outputs the score vector u of the current state. (t) =f Q (S (t) ), f Q (·) represents an agent model for reinforcement learning; WITH' (t) =σ(Norm(W Z WITH (t) +b Z )) (6) C′ (t) =σ(Norm(W C C (t) +b C )) (7) Among them W Z With b Z W C With b C and Norm represent the learnable parameters and biases of the embedded Z subnetwork, the learnable parameters and biases of the cluster center C subnetwork, and the normalized network layer, respectively; Z′ (t) and C′ (t) These represent the outputs of the corresponding sub-networks; the agent network concatenates the two learned parameters to calculate the score for the current state, as shown below: u (t) =σ S (W S (Concat([Z′ (t) ,C′ (t) ]))+b S ) (8) Where σ S W S b S Concat and represent the softmax function, which outputs the learnable parameters and biases of the network, and the concatenation function, respectively. Through the combination of the above, the agent fuses the state of the original data with the state of the cluster center, and performs iterative learning according to different actions to achieve automated cluster number determination. S206. Establish the action strategy in the graph clustering method of deep reinforcement learning. DRLGC establishes a greedy strategy for reinforcement learning based on formula (9). The greedy strategy shown in formula (9) evaluates different cluster numbers and makes intelligent decisions during training, where the greedy parameter ε increases with the number of iterations. The `random()` function represents randomly selecting an action from the action space. S207. Establish a state transition strategy in graph clustering methods for deep reinforcement learning. After the agent makes an action based on the current state, the environment's current state S... (t) It will transform into S at the next moment. (t+1) ; S208. Establish the reward function in the graph clustering method of deep reinforcement learning by minimizing the distance between each node and its corresponding cluster center. To improve inter-class similarity, the distance between different cluster centers is maximized. To improve category discrimination ability, the reward function is expressed as follows: By maximizing the reward function, the agent can make the most accurate judgment.

5. The threat detection method for low-quality labeled data environments according to claim 4, characterized in that... The key elements of the reinforcement learning decision-making process include environment, state, agent, action, state transition, and reward function.

6. The threat detection method for low-quality labeled data environments according to claim 2, characterized in that, Step three includes: S301. Establish a DRLGC model training method that includes training of a deep clustering network and training of an agent. The deep clustering network is trained in a self-supervised manner, and the agent adopts an experience replay training strategy, establishing an experience buffer by using a four-tuple of current state, current action, current reward, and next state. Where t s This is the buffer's initial iteration state, l t The buffer limit width is [value], and the number of iterations in the deep clustering network is [value]. Based on the collected empirical buffer, the loss L will be minimized. Q To train the agent and make the agent f Q Take the optimal action; the loss function is expressed as follows: Where η is the discount factor; where the first two terms in formula (13) represent the calculation of the target value, including the discount estimate of the immediate reward and the future optimal value, and the last term represents the value prediction of the current action, which is used to compare with the target value. The overall goal of the loss function is to make the predicted value close to the real action value by minimizing the square of the temporal difference error. S302. Train the DRLGC model to obtain the final number of data clusters. Deep clustering models are used to simulate changes in the current environment, state S (t) It uses a deep clustering model based on the current action Generate; Reward R (t) The result is calculated based on each node and its corresponding cluster center, and then further based on the reward R. (t) The reinforcement learning agent is trained with the experience pool to establish the DRLGC model; The depth map clustering method integrates data features and their neighbor information to obtain the embedding representation of the t-th iteration using a spatial feature fusion approach. Where d′ represents the number of features after dimensionality reduction, and f E The encoder in the depth map clustering algorithm is a combination of the encoders from AE and GAE; in the action Clustering results obtained under the guidance of deep clustering loss. Furthermore, the obtained clusters are mapped to a low-dimensional space, preserving the similarity of data points within clusters and the differences between clusters, thus obtaining the cluster center representation. Using data embedding to represent Z (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) The status will include node information and cluster information of the original data; Based on the established state information, DRLGC further establishes a reinforcement learning agent f. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. The greedy strategy shown in formula (9) is used to evaluate different cluster numbers and make intelligent decisions during training; the final number of clusters is obtained through multiple iterations.

7. The threat detection method for low-quality labeled data environments according to claim 2, characterized in that, Step four includes: S401. The original data undergoes fine-grained classification and labeling. For existing low-quality labeled data and the obtained clustering results, the fine-grained labeling method based on clustering-label co-mapping follows these guidelines: (1) When the clustering results It contains only data of a single known category Y j When (j=1,…,m)∈Y, a deterministic mapping relation is established, i.e., P i Belongs to category Y j Y represents the low-quality labeled data of the original data, which contains only partial labels and may not be accurate. This represents the final calculated number of clusters; (2) When there is a missing category, there exists Then determine P i For potential new classes; (3) In a coarse-grained scenario, there exists P i With P j When all clusters contain the same single-category label, it is determined that a coarse-grained label exists, and a new label is created for the cluster set that contains the fewest instances of that single-category label. (4) In mixed category scenarios, i.e., clustering P i When there are two or more labels, this embodiment learns from the existing label data through a decision tree and performs clustering P. i The samples in the data are relabeled; the clustering-label co-mapping operation is repeated multiple times to make the number of data types obtained by label matching as equal as possible to the number of clusters calculated by DRLGC; S402. Use the encoder f that has already been trained in the DRLGC model. E The detection model is trained using the Decision Forest method to complete threat detection.

8. The threat detection method for low-quality labeled data environments according to any one of claims 1-7, characterized in that, The method integrates data features and their neighbor information to obtain the embedding representation Z for the t-th iteration using a spatial feature fusion approach. (t) In action Clustering results are obtained under the guidance of deep clustering loss, and the obtained clusters are mapped to a low-dimensional space; the similarity of data points within clusters and the differences between clusters are preserved to obtain the cluster center representation C. (t) Z is represented using data embedding. (t) With cluster center C (t) To establish the state S for the t-th iteration (t) ={Z (t) C (t) The state will include node information and cluster information of the original data; based on the established state information, a reinforcement learning agent f will be further established based on a graph clustering model of deep reinforcement learning. Q (S (t) ), in the reward function R (t) With agent loss L Q Under the guidance of [the relevant authority], a determination of the current number of clusters is obtained. Finally, a greedy strategy evaluates different clustering numbers and makes intelligent decisions during training; the final number of clusters is obtained through multiple iterations.

9. A system for implementing the threat detection method for low-quality labeled data environments as described in any one of claims 1-8, characterized in that... include: The data acquisition module is used to collect network traffic, represent it as a feature matrix, and use it as training data. The graph clustering building module is used to build graph clustering methods based on deep reinforcement learning; The detection model training module is used to train and establish a detection model using the collected network traffic. The network threat detection module is used to tag data and detect network threats using existing information.

10. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method of any one of claims 1-8.

Images